On-Call Me Maybe

If you loved On-Call Me Maybe, check out OCMM co-host Adriana Villela's new podcast, Geeking Out, which she produces alongside her daughter, Hannah. Geeking Out is the spiritual successor to On-Call Me Maybe, and it features the same topics that you've come to know and love from OCMM, along with a fabulous guest lineup, including Hazel Weakly, Tim Banks, Abby Bangser, and Jennifer Moore, just to name a few. Geeking Out launched its first episode on September 5th, and so to give you a taste of what Geeking Out is all about, you can check out the first episode on this feed. If you love what you hear, please subscribe to Geeking Out wherever you get your podcasts. Or, find Geeking Out on YouTube at geekingout_pod.

About the guests: Adnan Rahić Adnan is an author, failed startup founder, and ex-Free CodeCamp leader. Adnan has pent the last 5 years building open-source developer tooling in the Observability and Big Data space. He is currently leading the Developer Relations efforts for Tracetest.io. Part time professional boxer and body builder.   Oscar Reyes Oscar is a problem-solver, driven and motivated person. Oscar has 8 years of experience building backend cloud native systems, front end pixel-perfect applications and automated CI/CD pipelines. He is now the Lead Engineer at Tracetest.io.   Ken Hamric Ken has 30+ years experience in developing software products, with a focus on enterprise testing solutions. He founded CrossBrowserTesting.com in 2008, which was acquired by Smartbear, a leader in the software testing category, in 2016. Ken stayed four years at Smartbear as VP of the web and mobile testing platforms. His most recent startup, Tracetest.io, is leading the way in utilizing observability data to enable deep integration testing for cloud native applications.    Find Adnan Rahić on: Twitter LinkedIn GitHub Find Oscar Reyes on: Twitter LinkedIn GitHub Find Ken Hamric on: Twitter LinkedIn Find Tracetest on: Twitter LinkedIn Find us on: On-Call Me Maybe Podcast Twitter On-Call Me Maybe Podcast LinkedIn Page On-Call Me Maybe Podcast Mastodon On-Call Me Maybe Podcast Instagram On-Call Me Maybe TikTok On Call Me Maybe Podcast YouTube Channel Adriana’s Twitter Adriana’s Mastodon Adriana’s LinkedIn Adriana’s Instagram Adriana’s Bluesky Ana’s Twitter Ana's Mastodon Ana’s LinkedIn Ana's Instagram Ana’s Bluesky Show Links: Tracetest OpenTelemetry Semantic Conventions https://thenewstack.io/trace-based-testing-the-next-step-in-observability/ https://thenewstack.io/where-does-trace-based-testing-fit-in-the-testing-pyramid/

About the guest: Hannah: Hannah is a high school student living in Toronto, Canada. She’s also the daughter of OCMM co-host Adriana Villela. She loves showing off her artistic side, whether it’s through baking, sewing, painting, or knitting. You can see her creations on Instagram. Hannah has been bouldering since she was 3, and still enjoys going to the bouldering gym with her parents. Although both of her parents work in tech, Hannah would rather not spend her time sitting at a desk and staring at a computer all day. That’s why she’s planning on becoming a dentist when she grows up.  Miss Unicorn: Miss Unicorn is Hannah’s trusty Squishmallow friend. She currently lives in Adriana’s home office in Toronto, alongside fellow Squishmallow buddies Barry the beaver, and Gerry the giraffe. Find our guest on: Instagram Find us on: On-Call Me Maybe Podcast Twitter On-Call Me Maybe Podcast LinkedIn Page On-Call Me Maybe Podcast Mastodon On-Call Me Maybe Podcast Instagram On-Call Me Maybe TikTok On Call Me Maybe Podcast YouTube Channel Adriana’s Twitter Adriana’s Mastodon Adriana’s LinkedIn Adriana’s Instagram Adriana’s Bluesky Ana’s Twitter Ana's Mastodon Ana’s LinkedIn Ana's Instagram Ana’s Bluesky Show Links: Dall-e Additional Links: Video Episode  Miss Unicorn’s Siblings 

About the guest: Hannah: Hannah is a high school student living in Toronto, Canada. She’s also the daughter of OCMM co-host Adriana Villela. She loves showing off her artistic side, whether it’s through baking, sewing, painting, or knitting. You can see her creations on Instagram. Hannah has been bouldering since she was 3, and still enjoys going to the bouldering gym with her parents. Although both of her parents work in tech, Hannah would rather not spend her time sitting at a desk and staring at a computer all day. That’s why she’s planning on becoming a dentist when she grows up.  Miss Unicorn: Miss Unicorn is Hannah’s trusty Squishmallow friend. She currently lives in Adriana’s home office in Toronto, alongside fellow Squishmallow buddies Barry the beaver, and Gerry the giraffe. Find our guest on: Instagram Find us on: On-Call Me Maybe Podcast Twitter On-Call Me Maybe Podcast LinkedIn Page On-Call Me Maybe Podcast Mastodon On-Call Me Maybe Podcast Instagram On-Call Me Maybe TikTok On Call Me Maybe Podcast YouTube Channel Adriana’s Twitter Adriana’s Mastodon Adriana’s LinkedIn Adriana’s Instagram Adriana’s Bluesky Ana’s Twitter Ana's Mastodon Ana’s LinkedIn Ana's Instagram Ana’s Bluesky Show Links: Macaron Folding (baking term) Meringue Convection oven Baking soda vs baking powder Cracked macarons Silpat baking mat Additional Links: Video Episode  Miss Unicorn’s Siblings 

About the guest: Adrian Cockcroft is a technologist and strategist with broad experience from the bits to the boardroom, in both enterprise and consumer-oriented businesses, from startups to some of the largest companies in the world. He is equally at home with hardware and software, development, and operations. He’s best known as the cloud architect for Netflix during their trailblazing migration to AWS and was a very early practitioner and advocate of DevOps, microservices, and chaos engineering, helping bring these concepts to the wider audience they have today. Find our guest on: Mastodon Medium GitHub Find us on: On-Call Me Maybe Podcast Twitter On-Call Me Maybe Podcast LinkedIn Page On-Call Me Maybe Podcast Mastodon On-Call Me Maybe Podcast Instagram On-Call Me Maybe TikTok On Call Me Maybe Podcast YouTube Channel Adriana’s Twitter Adriana’s Mastodon Adriana’s LinkedIn Adriana’s Instagram Adriana’s Bluesky Ana’s Twitter Ana's Mastodon Ana’s LinkedIn Ana's Instagram Ana’s Bluesky Show Links: Nubank OrionX Assembly language ChatGPT BASIC Lunar Lander Sun Microsystems ALGOL Pascal R S-PLUS AWK Go HPC Cluster Amazon Web Services Netflix Reed Hastings (Netflix co-founder) Storage Area Networks Oracle Hadoop Cassandra Adrian @ QCon London ‘23 Chaos Engineering Adrian + Ana @ AWS re:Invent 2018: Breaking Containers: Chaos Engineering for Modern Applications on AWS  Catchpoint Monitorama speaker schedule Harness Linux Foundation Cloud Native Computing Foundation (CNCF) Green Software Foundation CNCF’s Kepler European Union’s Climate-Change Tax Plan Liz Rice KubeCon EU ‘23 - Amsterdam Adrian on Platform Engineering Spinnaker The Value Flywheel Effect by David Anderson Wardley Mapping AWS CDK Patterns Jeremly Daly Ampt AWS’s GameDay Pre-Accident Podcast Sidney Dekker Chernobyl Gluecon 2023 Lean Agile Scotland Additional Links: Laguna Seca Racetrack Alvardo Street Brewery Duckzilla from Duck Foot Brewing Netflix’s Journey to the Cloud   Transcript: ANA: Hey, y'all. Welcome to On-Call Me Maybe, the podcast about DevOps, SRE, Observability Principles, on-call, and everything in between. I'm your host, Ana Margarita Medina, and with my awesome co-host...  ADRIANA: Adriana Villela. ANA: Today, we're talking to Adrian Cockcroft, who's an Advisor in Tech. And we get to hear about all his experiences throughout these years. We're so happy to have you join us. ADRIAN: Thanks. Great to be here, and good to see you again. ANA: Definitely. Where are you calling from today? ADRIAN: So I live in California. A few years ago, I used to live in Los Gatos near Netflix and eBay, where I worked for many years. And then a few years ago, during COVID, we sort of did the, hey, we're going to retire and get out of the Bay Area thing and bought a house down about halfway between Monterey and Salinas. Anyone that knows racetracks, I can see Seca racetrack out of my window, and I can hear it when the cars are running. So we're right by a racetrack, which is part of my fun and games. ANA: Nice. My favorite breakfast burritos are in Salinas, and I'm a huge fan of Alvarado beer out there too. ADRIAN: Yeah, it's nice to be out of the Bay Area sort of craziness. I mean, there's still a bit of traffic here, and there aren't so many roads, but it's a much more relaxed place. And we've got a lovely place down here, and the house prices were a tiny fraction of what it would cost to have a nice place in Los Gatos. So that was the main thing. ANA: That is awesome. And for the second question, it's our traditional On-Call Me Maybe Podcast question, what is your drink of choice for today? ADRIAN: I'm actually drinking some water because it's about noon. But if it was a bit later in the day, I'd be drinking a gluten-free beer. There's a brewery that I really like called Duck Foot, duckfootbeer.com. They make a beer called Duckzilla. They make a stout. They make all kinds of interesting beers. And Stout Mask Replica is the name of one of their styles, which has a picture of Captain Beefheart on it.  ANA: [laughs] ADRIAN: And there's the "Trout Mask Replica," which is an album that some people might remember from the late '60s or something. Anyway, it's hard to find gluten-free beer. This is a brewery founded by somebody who has celiac. So all of their beers they don't make a big fuss about it, but all their beers have the gluten taken out. So the proper beer is made the normal way. There's an enzyme you put in that takes the gluten out. They do a full range of interesting beers. But they're in San Diego. They'll ship to California. If you're outside California, it's pretty much impossible to get.  ADRIANA: [laughs] ADRIAN: So every now and again, I go on their website, click buttons, and then a big box of beer turns up full of stuff for me that I can actually drink. Unfortunately, I like beer, but I got gluten intolerant a few years ago and now have to manage what I eat. ANA: It is really nice that we're getting more inclusion in the beer and alcohol space that now there's less sugar or gluten taken out or just the way that they're being promoted thinking about, hey, everyone has different dietary restrictions. ADRIAN: Yeah. And if you get a stomach ache every time you have pizza or drink beer, it's probably because you've got some gluten intolerance, and when you eventually just stop, you discover all the stomach aches went away and carry on with your life.  ANA: Just experiment. [laughs] ADRIANA: Cool. Well, we should probably get into some of our more techy questions. So a question that we love to ask all of our guests is, how did you get into tech? ADRIAN: I'm quite old, so this was a long time ago. [laughs] My dad was in technology, basically. He was testing anti-aircraft missiles back in the 1960s when I was born. I was christened on a missile test-firing ship whose motto was aim high. [laughter] So I thought that was...if you're going to have a motto, that's fine. And then, in the 1960s, he went to college and was programming a little bit. He did a mature degree in statistics when he was in his 20s and then got a job at a university as a statistics lecturer. So, basically, I grew up with my dad doing computing and statistics through to the time I was growing up.  And somewhere along the way, the school had a computer, and my dad used to sort of put things in front of me. He didn't really teach me stuff. He just left things laying around that I looked at. [laughter] Computer Weekly magazine was a weekly newspaper that was always around the house. And so it was always things like that. But I taught myself to code at high school in 1972. We had a teletype connected to a deck system 10, and it had BASIC on it. And I played Lunar Lander and tried to figure out how to write things in BASIC and a few other programming languages.  And then, when I went to college, I did physics, but I ended up doing applied physics and electronics, which there was a bunch of designing computers that could control physics experiments. It was kind of what they were trying to teach us. So I did Assembly Language and some C programming and basically embedded stuff. And then, the first job I got was at a place called Cambridge Consultants in Cambridge, UK. I was basically doing embedded real-time development full-time. That was my day job. I did that for about six, seven years, just writing code every day. Most of the code ended up burning it into a PROM and sticking it in a machine, leaning on the stop button while you press the start button in case the code went wrong, and all the motors went backwards or something. So we had fun with that. And then we were using Sun machines as development machines.  During that time, I ended up basically as a one-day-a-week sysadmin because there was a full-time admin looking after all of the faxes and things. And there was one day a week I basically looked after all the Unix machines as a part-time job. So this is the early days of DevOps since I was deving four days a week and opsing one day a week. It mostly consisted of making sure the backup tapes got changed for a few Sun machines.  And then, I joined Sun in 1988. They opened an office across the street, and eventually, I went...I wanted to know what they were going to do next. And basically, I talked my way into what you'd now call a solutions architect job though we called them systems engineers then. And I did that for about six years in the UK.  And then I ended up moving to the U.S. to be in sort of the central group that supported that function globally, so writing white papers and trying to get all the information in the field people needed to learn about the new machines as they came out. And various things happened after that, but that's kind of how I got into tech, basically, by my dad bringing home random computers and calculators and things as I was growing up. ADRIANA: That is so cool. You mentioned learning BASIC. BASIC was my first language. I have very fond memories of BASIC. How do you find the evolution in programming languages from the early days of writing BASIC code to the stuff that's out there now? As you reflect on that evolution over the years, what's your thought? ADRIAN: I mean, some languages I find easy, and others I find hard to figure out. BASIC was just like; I had no idea what I was doing, absolutely no clue. ADRIANA: [laughs] ADRIAN: You could type things into this computer. And it was a roll of paper going up the screen. And you just, I don't know, you could make this thing do stuff by typing stuff in. So that was pretty...I had no theoretical background of what was happening at all. At some point, I decided to learn ALGOL, which is a more structured language. And that was where I kind of got into structured programming a bit more. And then there's like Pascal kind of things, those sorts of languages.  And then the first job I had I was mostly programming in C. At some point, they gave me a terrible job with a really horrible language and development environment. And to sort of keep myself sane, I built myself a home computer and ported a C compiler to it. So I ended up rewriting the code generator for a C compiler as sort of a home project.  ADRIANA: [laughs] ADRIAN: It was a very small, think of a small C. But what the effect of that was is I ended up a much better C programmer because I now had a code generator in my head. I could look at the code, and I'd look at the C code, and I actually knew what it was going to do to the machine and much more. So actually, having spent a bit of time working on the guts of a language, it's very simple, but it gives you a much deeper understanding.  I stopped being a professional programmer just as C++ came along, so I never learned C++. And then Java I used a bit, but I was never really that happy with Java. It's just too much stuff there. It's just too complicated and wordy. And then, let's see; I learned S-PLUS, which is now R. So most of this analysis code I do is in R. And because I know R, I never tried to learn Python because if I need to do something, I do it in R.  Similarly, I never learned Perl because I could always do it in Awk or something like that. I learned Awk before Perl came along. And there are all these sorts of things where I know how to do with things, so I don't need to learn another thing or another way of doing it. When I was working for a venture capital firm in about 2014, everything started being written in Go, and it was all open source. So a startup would come in, and I'd start rummaging around in their repo on GitHub and trying to read the Go code, and I went, okay, I need to learn Go.  ANA: [laughs] ADRIAN: So I learned Go, and I built some pretty complicated things in it. I learned the language fairly well. For a few years, I was sort of just experimenting and playing around in it. So that's the last time I really learned a language. And I liked Go quite a lot. I like the fact that it's just go build, and it's done. And it's a very lightweight environment to get into. And I liked the fact that they haven't changed the language much. So I probably still know Go [laughter], although I haven't written code with it for quite a few years. [laughs] Most of the other languages have changed so much you actually don't know how to use them anymore. ADRIANA: That's a really good point. ADRIAN: So you sort of have...I think languages come along, and then they wear out. They keep adding features to them until nobody new can really learn them anymore.  ANA: Yeah, it's true. ADRIAN: They're just too top-heavy. And then you need a new language that's simple. That happens with a lot of technology. You get this sort of technology cycle where the big complicated thing is like the innovator's dilemma. It gets too big and complicated. And you get the new thing coming in underneath it, and it's just like a simple version of whatever it does, like all things. ANA: Especially with the adaptation of people being like, oh, this doesn't solve the needs I have in my organization. Or when we look at open-source work specifically, it's like, well, that company is not using the language anymore. So why are they going to continue contributing to it? And they moved on to something else. ADRIAN: I had the same thing with the cloud. When I was at Netflix, we were using EC2-Classic. That's like one big flat network. You just push a button, and a machine appears. There's none of this complex stuff with networking and all this really complicated stuff. So actually, I don't know how to use the cloud now. [laughter] But I did get a ChatGPT account recently. It generates code which looks plausible for stuff that might work in the cloud. But then I tried visiting some URLs that it brought up, and those didn't exist. So it invented some GitHub repos. [laughter] It sounded like it'd be really good if this thing existed.  ANA: Interesting. ADRIANA: Damn. ADRIAN: But I think if you're in a place that's very well documented and understood, and you're trying to understand it, it's probably got enough background information to actually give you good answers. But if you get off into the weeds to do something that's a bit marginal, it'll start making things up that are unplausible. So still playing around with it. But I figure now that the AI has learned all the languages, I can figure it out instead of using Stack Overflow.  ANA: [laughs] ADRIAN: The other thing I find is if I'm trying to build something complicated, eventually, I end up deep in the guts of some JavaScript, and I hate JavaScript. ADRIANA: [laughs] ADRIAN: It's like everything I do ends up stuck in some stupid JavaScript thing that I end up trying to debug something that doesn't work in JavaScript, and I wasn't trying to use JavaScript in the first place. ANA: [laughs] ADRIANA: It's funny because I am not a fan of JavaScript, either. And I'm sure...I'm sorry, listeners who love JavaScript. I just could never get the idea behind JavaScript. So I can completely relate. [laughs] ADRIAN: Yeah, it's all 1 + 1 equals 11. [laughter] ANA: As much as I liked writing it growing up, now it's definitely like, I'm so far away from it that I'm like, you know what? I'm okay without it. And then all of a sudden, I'm working on a project, and they're like, can you write TypeScript? And it's like, wait, why? [laughs] Why are we still here? ADRIAN: Yeah, TypeScript is probably a bit better. But there's a limit to how much time I...the other end of this thing was you said how to get into tech, but how to get out of tech; I'm not sure quite how I've got out. But last summer, I retired from my sort of day job. I was a VP at Amazon. There are all kinds of stuff you have to do at a big company. And it was really difficult to do podcasts because you get PR and have to be onboard and a whole bunch of other stuff.  So I'm really enjoying the freedom of being my own boss to do what I want. And I spend most of the time on vacation somewhere. And then in between, I do advisory work and try to be useful around technology and see if I can do a few conferences that I feel like that are in...sort of when I feel like it kind of basis. And I'm doing some advisory work in return for shares and little bits of consulting and paid speaking engagements and things. So that's my kind of how do you get out of technology? I'm sort of half out, but I've still got one foot in. ADRIANA: [laughs] I feel like you can never fully get out, especially if you really love technology. It's always going to find a way to find you, which isn't a bad thing. ADRIAN: Yeah, young enough to still keep working on stuff. My dad is still alive, but he's in his late 80s now, and he has Parkinson's, and he's getting more and more frail. And I asked him some statistics questions, and he said, "I've forgotten all that stuff." [laughter] It was long enough ago. ADRIANA: Technology languages have changed a lot since you started your career. And you found yourself in the cloud space at one point in your career. How did you get into it? ADRIAN: When I was at Sun in the early 2000s, about 2002 to 2003, we had a high-performance computing group that was looking at these big clusters that we used that were HPC, clusters of Linux machines. They're massively scaled Linux clusters. And we were looking at that. The other thing that we were looking at was building a machine which never went to market, which would have been so big that we wouldn't have been able to ship it. It was more than one rack in size. So you had to disassemble it and put it in several racks and then sort of glue them together to get your machine.  And we said, well, why don't we just leave it in the place where we built it and not ship it to customers and have them connect to it over the internet? Which was sort of conceptually, yeah, that's cloud. It made sense. And you're seeing that a little bit now with quantum computers. You really don't want a quantum computer in your data center. It's a monstrous thing. It's like having a nuclear reactor in your data center. It's full of weird plumbing and stuff. So you'd want to access it over the network.  So we thought about it a bit, but Sun couldn't figure out how to sell directly to consumers with credit cards who just didn't understand that market at all. And when we talked to the CIOs, they didn't want cloud. They wanted to build bigger data centers. And then Amazon came along and figured out...only knew how to sell stuff on credit cards, so it got in via that route. And the CIOs, many of them still don't like cloud. It was sort of an end run around the ability of CIOs to control where all of the technology came from was that you could just buy it on a credit card. So that was the thing that I think bootstrapped cloud as a thing.  And then, in about 2008, 2009, I was at Netflix, and we were trying to figure out how to scale what we needed for streaming, which was growing vastly faster than the capacity we needed for DVD shipping. And we had a big outage in 2008. We could talk about outages if you like. It was SAN. Remember Storage Area Networks? ADRIANA: Mm-hmm. ADRIAN: We had a virtual SAN controller, and all our machines were connected to the SAN network, and then all the storage was out the back. And one of the SAN controllers decided to silently corrupt all of the blocks, every now and again, silently corrupt a block going through. And then Oracle decided all our disks were bad. Everything collapsed. And then we restored everything, and then it corrupted it again. And then, we rebuilt the entire system from scratch out of new hardware, but that took three days. So we were down for several days.  The outcome from that...but this is one of those things...you tend to see people having a huge outage, and then you go, okay, so what could we do differently? It's when management pays attention to the systems and the assumptions. And we said maybe we should not be trying to run our own infrastructure. And Reed Hastings, in particular, sort of said, "Could we go use this Amazon thing? Can we just rent it from somebody else?" And we went to Amazon, and Amazon said, "Come back in a year. We're not ready to do that."  ANA: [laughs] ADRIAN: This was in 2008.  ADRIANA: Oh damn. ADRIAN: So we came back in '09 and started with some stuff that wasn't customer-facing movie encoding and some Hadoop cluster stuff. And then, during the whole of 2010, is when we went and moved the entire front end to the cloud sort of one page at a time. It took about nine months. And then, in 2011, we moved the backend of Oracle onto Cassandra. So it was sort of roughly a two-year process to get what people think of as the Netflix streaming product.  There was still data center stuff and other things happening in the data center for quite a few years. But they gradually moved corporate IT and all the other billing and things like that out. It was a multi-year thing. But it was an interesting opportunity driven by a very visionary kind of management team that Netflix is not scared of trying something new, certainly at that time. It was like; we'll try something new. Let's see if we can use that as an advantage to get ahead of the competition. ANA: As I came on to learn about cloud computing, I was always impressed with Netflix's story of just jumping in. And, I mean, it was one of the first companies to do it and for it to kind of go well. And I guess I didn't realize also how long it had taken. ADRIAN: And the other thing was in about 2010, 2011, I switched. I was managing a team. I was managing one of the development teams, and I switched roles to being the overall cloud architect. And I started going out and trying to explain what we were doing. So since then, my job has been to try and explain what Netflix did in 2010 to people over and over and over again. That's my latest career, if you like. ANA: [laughs] ADRIAN: Because people still aren't doing some of the things. I've had some of my talks...I did a talk at QCon in March, which was something like what we learned and didn't learn from Netflix as a microservices retrospective. The video isn't up yet, but they'll post that at some point. It was one of those things where I was at Sun. I was a distinguished engineer, and I was out doing talks and training classes and lots of public speaking.  And then, when I was at eBay and Netflix, I really didn't do a lot of that for a year or two. And I said, "I could go out and start talking about what we're doing." And they said, "Oh, okay." They'd never seen me doing talks at conferences.  ADRIANA: [laughs] ADRIAN: I have a decade of experience of doing that. So I just went, okay, started going out, and made it more explicit. We did it as a technology marketing exercise. And the open-source program was part of that as well. We wanted to create a technology brand around Netflix, so everyone would think it was a high-tech company so they could attract better people to work there and have it be a sort of a halo over the movie brand, which is the main Netflix brand.  So it's actually difficult when you have these two brands, like, you've got a consumer brand, then you've got a technology brand. Something that eBay never wanted to do because it's a marketplace. They never wanted to be a technology brand, and they had a lot of good technology most people don't know about. And it's something that I see other companies doing a better job of nowadays is your company, one, having a big open-source brand around them. ANA: It is really awesome to see a lot of companies like, one, realizing that open source has helped them and that they want to contribute back. But then again, when they get to look at their own tech stack, and it's like, we've done really awesome work. Like, we could actually make this vendor-neutral and ship it out there and take all of our proprietary, confidential things and kind of just be like, hey, we build awesome tech. And there are good people here, and you get to know about the brand. But it is definitely a recruiting thing as well at play.  And since you mentioned the open-source work at Netflix, I think if it wasn't for someone at work, I would have never come across you and gotten a chance to meet you since I got my start in cloud computing next to chaos engineering. And, of course, we have Netflix's open-source Simian Army Chaos Monkey as one of those golden poster childs. ADRIAN: Yeah, that really helps. And then I was advising Gremlin while you were at Gremlin. We did a re:Invent talk together at some point. ANA: Yeah, that's right. Folks can watch our 2018 AWS re:Invent talk around chaos engineering. ADRIAN: Yeah, I was advising Gremlin before I joined AWS. AWS made me shut down all my advisory work because of conflict of interest, but I still got to pull you in for that one. And I'm now doing more advisory things, and that's part of this sort of new role I've got. I'm advising Catchpoint, which is an internet monitoring tool, more observability. You see them at Monitorama. And Harness is CI/CD. They've got a new thing for continuous resilience, basically adding chaos engineering to the pipeline. I gave a talk at Chaos Carnival last month.  And then Nubank, which is a big end user, Brazilian, next-generation banking company that's got all kinds of cool people working there like [inaudible 21:09] Michael Nygard. And so I like hanging out with them. So that was why I ended up advising them. ANA: Very nice. Another fun fact is that Adriana is actually from Brazil. [laughs] So when I saw that you're working with Nubank, I was like, hey, Adriana, look, [laughter] he's also working with like the largest bank out there.  ADRIANA: [laughs] ANA: As you're advising folks and you get a chance to not be carrying a brand next to you, is there anything that's exciting you right now about your work or the industry? ADRIAN: The thing I'm trying to push forward...I've got this position where I sort of have credibility, and I know people, but I'm a neutral party in the middle. So what I'm trying to do right now is get the cloud providers to agree on a real-time carbon standard. So what I want is...be it something like CloudWatch, or Prometheus, or whatever, I want there to just be a stream of one-minute interval metrics saying this is how much energy or how much carbon this machine, or this environment, or this pod, or whatever is using.  And right now, if you go and get the data from the cloud providers, it's monthly. You get a monthly total. And there are some ways of trying to guess what the numbers are from the billing records and things like that. But I think that what I'm proposing, I proposed this when I was at QCon London, what I'm proposing is that we should define what the standard is so that they will emit the same data, whether you're getting it from CloudWatch, or Prometheus, or whatever the...I forget whatever Azure has but whatever.  But they'd all have the same schema. They'd all be formatted the same way with the same meaning behind the data. So I've been on vacation for the last month. So I just launched the idea, and I'm just going to let it marinate. Sometimes you just launch an idea. And if you get too hot and heavy with it, you burn out. So I stuck the idea out there. I let it sit for a little bit.  And then, in the next month or two, I'm going to be trying to talk to people about is this possible? Can you get it together? Can we get people to agree that this could be done? And can I get at least one cloud provider to build it and then see if we can beat up the other ones to go and copy it and do the same thing? Rather than having very different data and interfaces and resolutions as the way we have it. Really, the data is all over the place.  The talk I gave at Monitorama last year was really about that because the way I think about it is that carbon should just be a metric like utilization, latency, throughput, carbon power. It should just be another metric that you have. And all of the monitoring tools should just be using it and passing it through. So you should see it in, you know, pick your favorite monitoring tool, you know, whatever, Datadog or something. It should just be there as a metric they can provide. And that's one of the reasons I'm talking to Harness and Catchpoint was to try and sort of push that forward with them. But then they can't build the talks if data isn't there. I'm sort of a vaguely neutral third party sitting in the middle. I know who's who at the different cloud providers, and I'm going to try and talk them into doing something or at least explaining to me why they can't, something like that. ANA: This also sounds like a great idea or opportunity to bring forward to the Cloud Native Foundation since they also work on standardizing some stuff across cloud. But I also know when projects are, like, early start, you kind of just need to do a prototype first. ADRIAN: Yeah. The Green Software Foundation is another Linux Foundation group where I've sort of pushed it there, and that we're going to sort of validate it there. There are definitely some things happening at CNCF, and I know the people there too. I was on the board there for a while. My team at AWS wrote the document that got us to join.  ANA: [laughs] Awesome. ADRIAN: So I had the budget to pay for KubeCon and all that stuff. So for several years, I was driving that from AWS' point of view. So yeah, I think I know the people there. There are actually so many projects going on that I was trying to just use ChatGPT to discover them all [laughs] and explain them to me.  ADRIANA: Yeah, there are a lot. ADRIAN: It sort of worked a bit. And some other people popped up and said, "Hey, there's this project and this project." There's a nice project called Kepler, which looks really interesting; that's a CNCF project. ADRIANA: Just going back to your carbon metric, I hope we get it to be part of standard metrics that are emitted because it comes full circle really nicely with...Ana and I were discussing in the first episode of the season. Ana, I don't know if you remember we were talking about sustainability and computing. And it's kind of cool because this is the last episode of the season.  But, I mean, even cooler is the fact that we are in an industry that is not great for the environment for the sheer fact that we are using computers. And when you're using data centers and cloud computing, that stuff uses a lot of energy. So to make carbon consumption, like, make it part of the conversation, I think is so, so valuable because I think we do need to be more cognizant of these types of things. Because the whole thing of, like, we've only got the one planet left, and if we have a way of tracking it, then we have a way of making it better. ADRIAN: Yeah. It's pretty complicated as well. There are a lot of things that aren't very obvious. If you look at a solid-state disk, it takes more carbon to make it than it will use in electricity in its lifetime. There's so much silicone in it, and the carbon is all in that, whereas the spinning disk is the other way around. There's a 10-watt motor spinning the disk all the time. So it's storage, and you put it on tape, and it's basically zero carbon. And then the other thing is if you get green energy everywhere, you're not done. In a few years' time, it's pretty easy to get the green energy. Progress is going really well there. But then you've got all the concrete in the buildings, which buildings are much bigger emitters of carbon. It's about, I think, 40% of the carbon emissions of the world is in buildings, building, and operating buildings, whereas IT is a few percent and in transport tens of percent, and things like that.  So if you're in a company like Amazon, most of Amazon's carbon footprint is deliveries. It's the aircraft and the packages and all the shipping. The AWS piece is a smaller part. If you can use more compute to optimize your real carbon footprint and your product carbon footprint, you know, moving at... we call it moving electrons and moving atoms. In the IT industry, we're moving electrons around. Electrons are much easier to move around than atoms. If you're moving physical things around or making stuff, or mining, or whatever, or transport, or buildings, that's where the carbon really is. So you have to think about your supply chains, and this is starting to come along. I saw an announcement today that the European Union is now going to start taxing the carbon footprint of things that they import. They've just finally agreed to that rule. It takes a while to kick in. But you can't just say, oh, they are only doing that in Europe; it doesn't matter because we're in the U.S. If you want to export something to Europe, you're going to be paying a carbon tax on it in the future.  ANA: [laughs] ADRIAN: So whatever they do in Europe will eventually spread around the world, even if the U.S. is a few years behind what's happening there. So, interesting times. ANA: It's always neat to see what Europe is doing around reliability or sustainability because they seem to pass laws a few years before the U.S. and it's like, oh yeah, they're doing it. Like, let's give it another five years, and America will kind of hop on. ADRIANA: Yeah, it was very cool being in Amsterdam and seeing how much more environmentally conscious they are compared to many cities in North America, and I don't mean just from a biking perspective. You will find different bins for different things like for your compost and your different types of recycling and all that stuff. And we're not even there. I go to a hotel room in the States, and there's just a garbage can. I go to a hotel room in Canada or in other parts of the world that are a little more environmentally conscious, and you'll find a recycling bin in the garbage can. So it's quite interesting.  ADRIAN: Yeah. I think I saw Liz Rice actually cycled to KubeCon from London or wherever she lives somewhere in the southeast.  ADRIANA: Damn. ADRIAN: She actually was posting about it.  ADRIANA: That's awesome. ANA: That's awesome. ADRIANA: One of the questions that I wanted to ask you is, since you've gotten a chance to see all of the cloud movement and be part of it, where do you see DevOps and SRE going in the next five years? ADRIAN: I did a blog post on platform engineering, which was sort of related to this [laughter] latest hot word, and then somebody...was it Jeff Cessna* [SP] who was muttering about this? He basically said something like, well, you can sort of divide the world into the enterprise world and what we used to call scale digital natives, if you know what I mean, like Netflix, Uber, companies like that grew up in this world. And they are organized very differently, and they behave differently. There are a few things that maybe don't fit neatly into that category.  But what we did in the scale digital natives is we just combined the development and operations stuff. We didn't create a separate team to do it and call it the DevOps team. We just taught our developers to operate and automated everything.  [laughter] ADRIANA: Oh my God. ADRIAN: And built all this tooling so that you wrote code, you hit check-in, and then you waited 15 minutes, and it was running. That was it. You didn't have to go -- ANA: Assuming Jenkins doesn't break.  [laughter] ADRIAN: Well, yeah. We did break Jenkins. But Netflix built a thing called Spinnaker because we were doing so much Jenkins abuse, basically.  ANA: That's great. ADRIAN: So somebody's there making the pipeline work. But you should just check code in, and it should just appear in production. You shouldn't be talking to operations to make that happen. So that was kind of the thing. And then we put all the developers on-call so that if your thing broke, you'd just shipped it that day, that you didn't have a meeting to explain to somebody else how it worked. So you just were on-call for anything that you'd done.  And then, of course, for people to get off-call, they had to have somebody else that also knew how it worked. So that kind of got people to do more pair programming or at least code reviews with somebody else. Like, whoever was on-call for that shift for a few days would be code reviewing all of the changes that went out to production that they didn't make themselves. So you ended up with sort of a buddy system where there were at least two people that knew how everything worked. And all we did to get there was put people on-call. Developers don't like to be on-call, but if it goes wrong, we're going to call you anyway.   ANA: That's true. [laughs] ADRIAN: So you may as well admit that upfront, and we put you on-call. And if you're changing it every day, like you could, then you didn't have a handoff meeting to explain how it works. There's no TOI, you know, transfer of information meetings. And you didn't write a huge amount of documentation about the way it works today versus yesterday. And there are probably five different versions running in production because you've got all these different test versions and old versions.  And it's complicated. You need the person that currently has their head around that little part of the system, which is why it's a microservice. It gives you a nice boundary. I know these bunch of microservices, and if something goes wrong with one of them, I know who to call because I know who built that service. So that was the way we did DevOps. But what happened in enterprise was they had an ops team, and then they found they couldn't hire ops people unless they called them DevOps. So they bought them a copy of Chef or something and started to call them DevOps. And then, The SRE book came out, so then they started hiring SREs and bought them some different tools. And now they're calling it the platform team and buying some more tools or whatever.  But it's sort of roughly the same team. And it's a separate team. And the developers are on one side. And there are more operators on the other side. And they didn't really solve for some of the problems that you really had where the developers didn't understand what it meant to operate something. I've always bridged the two. Like I said, I was a one-day ops person and a four-day dev person back in the 1980s, so I count that as DevOps. But I think that that's my sort of take on it. Then, where's it going? I think the interesting thing about where it might be going...because you get this quote from William Gibson that "The future is already here. It's just not evenly distributed." And I've been living that for many years. I'm still explaining things that we did ten years ago at Netflix to people.  The sort of bits of the future I've seen, there's a book called "The Value Flywheel Effect" by David Anderson. And it's largely about what they did at Liberty Mutual, which is an old company, but they got to go incredibly fast. It's a serverless-first approach, a lot of Wardley mapping to figure out what to do. And they ended up with the ability to deploy things in hours, completely new products, not a new version or a push some functional addition. They built a completely new product from scratch and delivered it tomorrow. And most people don't understand how you could do that, like, a new insurance product.  Okay, I have an idea for a product; okay, next day, we've built it. It took us a few hours. We've built it. We've shipped it. It's running now. We want to change it, okay, we'll change it again. They're going at a pace that's just unreasonably fast. And I think that that's kind of taking serverless-first taking serverless to the limit. There are lots of CDK patterns in there. In fact, cdkpatterns.org or.com, I forget which, was created by them. So they've just templated everything. If you want to build, most of the things you'd want to build are on common patterns. So that's one thing.  The other thing is, you know, this whole lot of fuss recently about ChatGPT. Something that you'll be able to do is have a conversation about the code you want to build, and it'll end up being built. That's where we're going. You don't need to know the language. You need to know what it is you're trying to solve for. And it's a conversation like you have a conversation with your business person about whatever, whoever wants to build the thing, the product manager, then you go and try and build it.  Effectively, the product manager needs to know a bit more about development, how development might look. But I think we'll end up in a place where most things are being managed and modified by having conversations with these sorts of AI bot things that are doing it. So that's, I think, where we go. ANA: It is interesting to think about how ChatGPT/AI is really going to transform the way we do platform engineering or build applications, like, the scaffolding aspect of it of your organization has these little blocks, and you just kind of get to merge them in together to get up and running as needed. ADRIAN: Yeah. And there are things that are too complicated. Like Kubernetes is too hard for any one person to understand. I think Java, the language, the entire environment around Java is too hard to understand. And it's used absolutely every day. ADRIANA: [laughs] ADRIAN: There are too many pieces to it. And the cloud, there are too many services on AWS for everyone to understand. But your chatbot is the thing that can understand all of them. So it's more about, well, I want to do this, or I'll pull these different services you've never heard of that actually do that and put that together. So I think that's where we're going.  Jeremy Daly, who talks of serverless patterns, is well known for that. He's got a startup called Ampt that I'm also helping advise. And with Ampt, you have a chunk of code, and you annotate a little bit. Then you say deploy, and it says, oh, it looks like you have an endpoint that is going to take an input request, so I need to create a web front end. And it just provisions everything around it. That's kind of the idea.  ANA: Whoa. ADRIAN: And I think if you take that and then you teach ChatGPT stuff around it, there's something there where you basically describe what you want, a little bit of annotation here and there to give it some hints. And maybe you don't even need that. It just looks at the code you've written and says, oh, this is a web server, so I'd better stand it up as a web server and open the right ports and put a firewall on and all these things.  And every time you deploy it, it can say, oh, there's a new service now. I'll use the new one. It's going to be different each time you deploy. So it's not the same as just sort of automatically generating a load of YAML and then getting stuck with editing it. It sort of takes all that out.  ADRIANA: That's very cool.  ADRIAN: And it's sort of a side effect. Like I said, back in the day when we first started using AWS, I did know how to use every AWS service there was because there weren't enough of them to confuse me. [laughter] And now -- ANA: You definitely can't do that in 2023. [laughs] ADRIAN: I am a pretty useless developer in terms of cloud development now. I'd actually have no idea how to get anything running. I'd have to use something else to do it. ANA: Yeah, I'm down to a select few AWS services that I'm like, I can still use you. There are too many services that are coming out, and I can't keep up. ADRIAN: I think the last time I actually was hands-on in an account was there's some competition thing they run at re:Invent where everyone gets in there, and they have little quizzes where they have to work through. ANA: GameDay.   ADRIAN: Yeah, one of those things. I joined one of the teams and was trying to help figure out how to solve for...it was something about unicorn rides or something like that. ANA: Unicorn Rentals.  ADRIAN: Unicorn Rentals, yeah. [laughs] ANA: Yeah, I think it's Unicorn Rentals. I think they've been running with the same software stack, and they've just been adding a lot more challenges. But it's always really fun to go to re:Invent and see how it's evolved every single year. Well, as we're getting ready to wrap up the episode, we wanted to ask if you have any practical advice for our listeners today. ADRIAN: I guess a few sort of larger-scale things. One of the things I've done throughout my career is share a lot. And I found that I became the expert in stuff by sharing what I was sharing answers to things. So you can kind of be the expert and say, "You have to come to me to get the answer," and try to lock it down so you can be the only expert. Well, that causes other people to go to other experts or set themselves up.  What I found was...I read a book on Solaris performance tuning back in the '90s that was all of the answers to all the questions that I'd seen over and over again, and blogging more recently, and all the presentations I do. So if you've half-figured something out, try to write it down. It's going to help you figure it out better. And then, if you share it and discover whether it's right or not, other people will give you feedback on it. So stick your neck out in public. It's worth, you know, even if you're wrong, you just go, oh, okay, there's a better way of doing that.  So that's kind of to encourage people to not get too hung up on imposter syndrome or whatever; just try things out. After a while, you get more confident because you...it's like standing on stage. The first few times you do it, you're scared. After you've done it a long time, it's no big deal. You get, oh yeah, there are 10,000 people in the audience, fine, okay. I'm just going to say whatever I say. It doesn't matter anymore.  But there's a sense of building up that confidence for sharing what you know is sort of a good way to build a personal brand that will get you the next job. It'll make the industry better, things like that. And then the rest of it is just trying to find a little gap where people don't see...if some people seem to find things difficult and you find them easy, that's probably a little space to go build some expertise. Most of my career has been trying to find a path like that, which means I'm now -- ANA: Experiment and share. [laughs]  ADRIAN: Yeah. The trouble is you end up...I have this problem. I'm not a typical thing, right? I'm not a VP of engineering. I'm not a developer. And I have a weird mixture of backgrounds. I've worked in marketing; I think probably more than I've worked in engineering, reporting to the CMO at Amazon for AWS for a while. I'm sort of branded as a technologist, but I've been able to leverage that into something that's much more communicating about technology really rather than anything else.  So anyway, that's what I've found. I'm not sure if that's useful. One of the things is just thinking about chaos engineering and where that's going. I think it's important to understand how computers are controlling more and more of what we do. And they're getting more involved in controlling our physical lives, and there are more safety-critical things happening. So I think it's important to understand safety.  And there's...Todd Conklin has a really good podcast called The Pre Accident Podcast that I listen to. I've been listening to it for years. I've actually been on it a couple of times to talk about chaos engineering. And Sidney Dekker. There's a whole bunch of other books and authors there. It's really important to understand what's happening out in the physical world because a few years later, you're implementing some software that's actually modifying or controlling something that's going to be safety critical.  And there's a bunch of principles there and just trying to make sure the usability...the usability of things when they're in failure modes is probably the weakest thing I see, and that's why you need chaos engineering. If you don't test it, you don't know what it's going to do when it's unhappy. And it will probably -- ANA: It's going to break on its own. ADRIAN: It will break on its own, but it will break in a way that confuses you. And then, when you try and fix it, you'll make it worse. And you see that over and over again. That's pretty much what happened at Chornobyl and all these other sorts of big runaway events. It's like operators got confused, and then they pushed all the wrong buttons and pulled their own levers and rebooted the wrong machines, and everything got worse. So that's kind of the you have to sort of practice how do you fix it when it's unhappy? And how do you get it to fix itself so it becomes more stable? ADRIANA: I think it's really good advice because I think sometimes, as developers, we're almost too scared to push our systems to the limits because it's like, okay, I got it working. Don't anger the code, right?  ADRIAN: Yeah. ANA: [laughs] ADRIANA: And we have to get over that fear because otherwise, weird things are going to happen in prod, and we will be completely unprepared, and it'll be worse.  ADRIAN: Yes. Like test to pass or test to fail, right? ANA: Yes. ADRIANA: Mm-hmm. ADRIAN: Like, that's really...and the whole test-driven development, which people should probably be...one of the things actually...this is interesting with the AI chatbots as well. If you write a test and tell your chatbot to write code that passes this test, they've got something they can work against. And they can run that code, and you say, no, this is the error message I got. And you've actually...if you do test-driven development and get the chatbot to do the rest of the work, right? That's a lot of, I think, where we're going to end up.  What are the things this has got to do? And then what happens when it fails? How do you want it to behave? And that's really this sort of continuous delivery and having a...in your delivery pipeline for a developer, having chaos engineering be part of that pipeline, rather than something operations does like the data center failover that's supposed to be done every year. So move it from being part of an annual exercise to being something that's part of the continuous development process. That's what Harness is, is a CI/CD company. They just added that capability. They launched that last month. So those are the sorts of things that I think are interesting right now. ADRIANA: That's awesome. Just before we wrap up, I know you've got some upcoming talks. So, do you mind telling us what some of them are? ADRIAN: I'll be at GlueCon in Denver in May, Speeding up Innovation and the Tipping Point. So this is generally talking about the sort of nature of innovation and the way that we see these different things moving. And that whole conference is really turning into...there's a lot of talks there around this sort of ChatGPT, like, what's that going to do?  The whole conference is one of those ones that tend to follow the latest bleeding-edge stuff. And so GlueCon doesn't have a fixed theme. The theme is whatever is new this year, and they just jumped on it. So the agenda is full of discussions around what is the impact of this sort of AI-driven sort of development operations stuff? So that's going to be interesting. And then I'm going to be at Monitorama. I'll see maybe both of you there. ANA: Yes. We'll be there.  ADRIANA: Yeah, my talk's right after yours. [laughter] ADRIAN: I came up with this really cool idea. I called it a tale of two histograms. It was the best of response times, it was the worst of response times, which is the opening line from "A Tale of Two Cities" by Dickens. And somehow, I've got to get the famous quote from the end; it was a far, far better thing, something or other. I'm going to see if I can work that into my talk somewhere.  ANA: [laughs] ADRIANA: Cool. ADRIAN: But anyway, I've been messing around with histograms of response times. And I had this very geeky idea for a talk, which is mostly a bunch of code in R that I've been fussing with. And then Jason said, "We'll make that the opening talk." And -- [laughter]  ADRIANA & ANA: No pressure.  ADRIANA: No pressure. [laughs] ADRIAN: All right. So I got to spend some more time coding to get that a bit more sorted out in some better, slicker talk if it's going to be the opening one. And then I think I'm going to be at Lean Agile Scotland in Edinburgh in September and then maybe at an event in Athens in Greece in the end of September. So I'm generally accepting invites from places that are interesting that I'd like to visit. [laughter] That's my sense rather than...Since I don't work for anyone anymore, that's my kind of algorithm for deciding where I want to talk, other than places like Monitorama, which is really a tribe.  ADRIANA: [laughs] ADRIAN: It's a tribal gathering, and I like to be part of the tribe there. ADRIANA: That's awesome. I'm definitely looking forward to meeting you in person at Monitorama. That'll be awesome.  ADRIAN: Yeah. It's been very cool. ADRIANA: Thank you so much, Adrian, for joining us in today's podcast.  Don't forget to subscribe and give us a shout-out on Twitter via oncallmemaybe and Mastodon. We're on all the socials these days, I think. Be sure to check out [laughs] the show notes on oncallmemaybe.com for additional resources and to connect with us and our guests on social media. For On-Call Me Maybe, we're your hosts, Adriana Villela and... ANA: Ana Margarita Medina, signing off with... ADRIAN: Peace, love, and code. ANA: Whoo. [laughter] ANA: Yay.

About the guest: Thilina (known as "T") is a Site Reliability Engineer at Lightstep. T has had a varied journey in tech spanning from support, project management, and systems engineering, which has eventually led him to focus on the "people" side of Reliability Engineering. He is extremely passionate about communications and how that plays a huge role in improving the performance of teams and increasing the reliability of their systems. Find our guest on: Twitter LinkedIn GitHub Instagram Blog Find us on: On-Call Me Maybe Podcast Twitter On-Call Me Maybe Podcast LinkedIn Page On-Call Me Maybe Podcast Mastodon On-Call Me Maybe Podcast Instagram On-Call Me Maybe TikTok On Call Me Maybe Podcast YouTube Channel Adriana’s Twitter Adriana’s Mastodon Adriana’s LinkedIn Adriana’s Instagram Adriana’s Bluesky Ana’s Twitter Ana's Mastodon Ana’s LinkedIn Ana's Instagram Ana’s Bluesky Show Links: Lightstep Brittish Colombia - Watersheds Chatime Boba Cisco PowerPoint Leading SRE with Empathy Chaos Engineering Principles SLOs (Service Level Objectives) SLA (Service Level Agreement) P1 Outage (Production Issue Gradations) Transcript: ADRIANA: Hey, y'all. Welcome to On-Call Me Maybe, the podcast about DevOps, SRE, observability principles, on-call, and everything in between. I am your host, Adriana Villela. And with me, I've got my awesome co-host... ANA: Ana Margarita Medina. ADRIANA: And today, we are talking to Thilina Ratnayake, who works with us at Lightstep working as an SRE. Welcome. THILINA: Howdy, howdy ho. Hello. ADRIANA: So nice to have you. Now, first things first, where are you calling in from? THILINA: I'm calling from Vancouver, British Columbia, Canada. ADRIANA: Whoo. Awesome. We've got two Canadians in the house today. You are outnumbered, Ana. [laughs] ANA: Once again, I'm outnumbered. I think we're having too many Canadian episodes. I'm going to speak to some On-Call Me Maybe managers, which is us. So we're going to have to start talking. ADRIANA: [laughs]  ANA: I need more representation from just the United States of America, specifically California. [laughs] ADRIANA: Oh yeah, that's true. Well, we did just record today with someone from California, so... [laughs] ANA: Yes. True. True, true. Here and there, I'm outnumbered. [laughs]  ADRIANA: Yeah, here and there, you're outnumbered, either because they're from Canada or they're Brazilian. [laughs]  ANA: True. ADRIANA: So it wouldn't be proper On-Call Me Maybe tradition if we didn't ask you what you're drinking today.  THILINA: Sweet, yeah. I am drinking some pristine mountain water from one of three watersheds that apparently feed my city from the mountains of British Columbia. Yeah, this is grade A water that's been filtered through a Brita filter.  ADRIANA: [laughs] THILINA: So you know what? I'm just drinking really nice mountain water.  [laughter] ADRIANA: This is the best selling of water that I've ever heard. I'm like, I want some of that. [laughs] THILINA: Cool. ADRIANA: I've got Lake Ontario water, yes. [laughs] ANA: I thought we were going to get a Wikipedia summary of coordinates of where you can find these watersheds. I thought that's where this was going. [laughs] ADRIANA: Oh my God. Yes. For real. THILINA: I had it up on my screen. And I was like, oh, one of three watersheds. Oh, there are three. I don't know which one it's from. Well, [crosstalk 02:19] ADRIANA: Well, we're going to have to include this in the show notes now because our listeners are definitely going to be curious about this. [laughs] THILINA: And what are you all drinking? ANA: For me, today it's actually green juice, which is kale, pineapple, apple. And I think I put ginger and turmeric. So I'm trying to go back into some more veggies and fruits in my diet. ADRIANA: That's awesome.  THILINA: Very nice. ADRIANA: Yes, I've got a glass of Lake Ontario prime H20. [laughter]  THILINA: Ooh, wow. ADRIANA: No bubble tea, sadly. But as we were talking before we were recorded, I was so happy that in Amsterdam for KubeCon, I found a Chatime, [laughs] which, for those who are not in the know, it's a bubble tea chain. It's an international bubble tea chain. And we happen to have one in Canada, well, more than one in Canada. We've got a few in Canada. So I was pretty excited. And I told the Chatime guy I'm like, "I have these in my country." [laughs] He probably looked at me like I was crazy. He's like, get away, woman. [laughs]  THILINA: So, a very important question, what was your order when you found that Chatime?  ADRIANA: Shoot. I think I got a lychee green tea.  THILINA: Ooh. ANA: Nice.  ADRIANA: Yes, yes. ANA: That sounds fancy and yummy. I love anything lychee. ADRIANA: I know, right? I'm a fan of lychee. Now, I can't tell if it's lychee or leechee because people correct me either way. ANA: I was about to say I call it leechee. I grew up calling it leechee.  ADRIANA: I call it lychee. I've got some Chinese friends who call it lychee. And then I've got other Chinese friends who call it leechee. One is always correcting the other. I'm like, which one is it? [laugher] So you never know. ANA: Hit us up on social media if you know how to pronounce this. [laughter]  ADRIANA: There's our sidebar. I guess we should get into our regular business. [laughs] So, Thilina, how did you get into tech?  THILINA: Well, it's a very interesting story that starts off with a news clipping on the back of my grade 10 high school classroom [laughs] because, in that class, that room was used for multiple classes, one of them being business 11 or planning 10. So I was in my Planning 10 class, which is a high school class that talks about how to do useful things like do your taxes, or get a utility bill paid, or decide your future in the world. And one of the assignments was, what do you want to do after you graduate? Like, what's your plan? Come up with three plans. And that was the assignment.  And me being a classic grade 10 high schooler, I had just left it off till the block before it was due. [laughter] And it was like lunch, and then the next class would have been the one where I had to present. I'm like, oh, I got to think of something. ADRIANA: [laughs]  THILINA: And then, looking to the side on this wall, was a newspaper clipping about Cisco, the telecommunications giant. And this clipping was someone in the business class had talked about how the future looks great for teleconferencing. In the future, we might have AI and holograms, so this is a good place to be. And I was like, well, I like computers. It's kind of cool.  ADRIANA: [laughs]  THILINA: And then, in the time between seeing the newspaper clipping and when it was due, I did a little research on how do I work at a tech company like Cisco? And it showed me a college program that I could do. It was like a diploma. So that was my plan A. And my plan B would be if I did my diploma and maybe a degree afterwards. I don't remember what plan C was. But I was like, cool, let's make a PowerPoint. Let's present it.  ADRIANA: [laughs]  THILINA: And that was it. So in a couple...a year happens, and I was like, what do I do after high school? And I was like, well, I've got this plan that I made last year. Why don't I just stick with it? And then I did. I did all of it to the letter. ADRIANA: That's awesome. ANA: [laughs]  THILINA: I did the diploma, and I did the degree. My first job was learning at Cisco while doing support there. So, yeah, unintentionally, a newspaper clipping set the tone of the next 10-15 years of my life. So yeah, the answer to your question is I got started with a newspaper clipping about Cisco. ANA: [laughs] ADRIANA: That is awesome. I love it. ANA: I love when it's always the most random experience that one has growing up that's like, this was the moment that the light bulb went off, and I was like, oh, I can actually get paid for various years to do this. [laughs]  ADRIANA: Yes, Defining Moment. Also, that sounds like a really interesting class. I don't think I ever took any classes on paying utility bills and stuff like that. That sounds awesome. [laughs]  ANA: Filing taxes. ADRIANA: Filing taxes. ANA: The fact that a school teaches you. Like... [laughs]  ADRIANA: I know. No one taught me that. My first tax filing I did manually, pencil and paper. [laughs] Why?  THILINA: Nice.  ADRIANA: [laughs] But that's a really cool start. And I love how the most subtle things, I don't know, they seem almost like passing at the time, and then they come back. It's like it was meant to be. So kudos for making that happen. That's amazing. ANA: And, I mean, I think it's great to also start asking you students and kids early on what they want to do for their career. I know for me, I figured it out in middle school, which I am extremely fortunate of that. But getting folks early in high school to be like, all right, you have electives. Start thinking about what you want to do three, four years from now. What makes you happy? What do you hate? Can you intern somewhere? Can you do a side project? Can you shadow someone?  THILINA: For sure. I think it's worth mentioning that part of that story involves me making a PowerPoint presentation. And the reason I bring this up is a lot of people talk about, like, oh yeah, I knew I always wanted to get into tech because when I was little, my parents got me this computer, and I started building programs. And that wasn't my story. For me, it was that my parents got me a Windows 95. And they didn't give me any games, but they got the PowerPoint suite. [laughter] So all day, I would just be making presentations on things.  And this is important because I got into tech, and my first job in tech was doing support, which is a lot of talking to customers. There was also presenting information. And a lot of people were like, "I just want to write code." I was like, "But I love talking about what we do and presenting it." PowerPoint, or the ability to present information, has served me so much more in life than being able to program. So all I'm trying to say is if you're curious about tech, it's not all about coding. There's a whole bunch of presenting and packaging information too, which I hope kids are taking away from it. ADRIANA: Yeah, there's something to be said about being able to convey the information to people, like, yes, I know this, but can I tell somebody else about this? That's an art form.  THILINA: Yep. ANA: One of the first bosses that I had was, like, we hire based on being a good person and a good communicator because we can teach anyone how to code. And it kind of stuck with me because I'm like, yeah, almost anyone can learn how to code as in if you put your head to it, and you do the work, and you're able to pick up on what's going on. But if you're not able to know how to be a kind person, how to communicate, how to work well with others, you're not going to be successful in any career you take part in, specifically technology or technology as we know it right now that's completely remote. ADRIANA: Yeah, I totally agree. I think there's this long-standing misconception that you don't need to be able to communicate to be in tech. And, I mean, I've been in situations where I've been surrounded by brilliant developers who cannot express themselves to me, and so, well, what's the point? Your brilliance is lost because you can't share your brilliance with others. THILINA: Yeah, that's really true. Communication is key.  ADRIANA: All right, yeah.  THILINA: That will be my tattoo if I get one. ADRIANA: [laughs] ANA: That would actually be a good one. I say that in personal and professional settings all the time. I'm like, communication is key; that's just it. Over-communicate whenever you can. THILINA: Yes, I really believe in over-communication. Over-communicating is key, like, when in doubt, over-communicate. That's the biggest takeaway I've had from working remote during COVID, not for nothing. That's just a thing that sticks in my head a lot. I have a lot of thoughts on how teams work well remotely, and that's a huge tenet of it.  ADRIANA: Oh, tell us more.  THILINA: One of the best teams I ever worked on was a hybrid remote team. And what I mean by that is we had a couple of members that were in office and a couple of members that were in the state, so like a different country, different geographical area. But the team never felt separate. I never felt like talking to person B was like, oh, they're not here. And the reason for that is because we communicated a lot to the point that our chat was never silent between 8:00 and 5:00 because there's always things going on.  And it wasn't just like work-related things. It was like, I'm getting some coffee or like...it doesn't have to be context specific. But communicating frequently when you're remote and conveying tone goes into helping the other things that you do. And I feel like that goes a long way when you're working remotely, especially if you start new and someone's trying to get a vibe check of everyone. If you are not next to someone, you can't shoulder-tap them. Hearing what people talk about frequently goes a long way for building that trust.  So I highly believe in over-communicating because you'll actually find the right norm easily versus not communicating enough. The onus is on other people to be like, okay, I guess I'll talk. What I'm saying is it's better to over-communicate. ADRIANA: Yeah, I agree with you. It's basically having a chatty Slack channel. Having a place where you're encouraging your team to shoot the shit, even if it's not necessarily work, right? THILINA: Yes. ANA: [laughs] ADRIANA: You're building up that camaraderie. Prior to coming to Lightstep, I managed two teams. I had 13 people working for me. And I was terrified because I'm like, oh shit, I've never managed a team remotely. So I made a huge effort to, like, okay, even though I have two separate teams, like, we had a Slack for each team, but we had a joint team Slack.  And always posted stuff, whether it's like team announcements or funny things, because I wanted people to get used to communicating with each other. And we'd have like a bi-weekly sync-up meeting where, you know, so that it's not an us versus them sort of scenario. So I completely agree the over-communication is super key in a remote setting. ANA: And I think you nailed it with the hybrid aspect. I mean, I think this came to fruition a lot during COVID where it was like, folks were still trying to figure out what was going to happen. It was like different countries and states were shutting down at different speeds.  But it's like, how do you make sure that everyone, no matter what work persona you have, actually carries a similar experience at your organization where it's like, we know how to collaborate really fast; we know how to move at similar speeds; we know how to get projects done? Or how do we make sure that all the tools that we have have collaborator access, that we're allowed to just kind of sync up quickly and be always on the same page? THILINA: Right. 100%? ANA: You mentioned that you got your start in support, but you're now working in SRE. I kind of wanted to ask you how was that transition for you? Why did you do it? How's it going? THILINA: Okay, so I have to start the story about what happened immediately after graduating, which is that immediately after graduating, a lot of my friends were going into software engineering roles. And for me, I was in this weird place of feeling like, oh yeah, I know how to program versus, oh my God, I don't know what I'm doing. And then I got [laughter] my first job working as a system support administrator. And for one reason or not, that place was not really the most supportive for a junior. And within two weeks, I was like, this is not for me; I'm out. I don't really know what I'm doing. This isn't -- ADRIANA: Oh no. [laughs]  THILINA: Yeah. So within the first month, I had gotten my first job and then left my first job. And I joined a different organization because I was like, maybe I'm meant to be in more of like front-end web dev, maybe that's my path. And this place was also probably not meant for a junior engineer because I remember my second week, I was super excited, submitted my first-ever PR for review. And then, it came back with 78 points of feedback or to change on a Trello card. And they're like, yeah, so go ahead and do this async or just let me know if you have any questions. And I'm like, Ooh, I have so many questions. Like, should I even be here?  ADRIANA: [laughs] THILINA: And, again, I was like, maybe this isn't for me. And so then I was feeling very hopeless and scared about what I was doing. And then my friend was like, "If you're not having fun doing dev work, come over to support. You know how to talk and type. You seem like you would have fun talking to people. And you're still in tech, so why not?" And I was like, sure. Let's do it.  So I jumped over into support and was there for like three years. And I was like, all right, now I feel confident in talking to people about tech. Let's get into some dev work. I did a whole bunch of internal internships. And I was like, yes, coding, that's the thing I want to do in microservices and networking. And then, I got into a team as a systems engineer. And it turns out that was the precursor to being an SRE because those systems we also maintained to ensure that they were running reliably. And it just so turns out that's what SREs do.  ANA: [laughs]  THILINA: So I was like, all right, SRE it is, and then just kind of magically dovetailed into there. So that was a bit more about that weird journey getting into tech and SRE specifically because it was a journey through IT, and front end, and support, and then all the way through systems to here, so yeah. ANA: What one or two qualities do you think that SREs need to learn from support engineering? THILINA: Oh, here we go.  ADRIANA: [laughs] THILINA: So here's something you learn during support is that the amount of time that the user or the customer has for you is very limited. And you're a good support engineer if you can get the information to them quickly in a way that they can understand quickly, but they can also refer back to very easily. So you're working with tickets. If your ticket takes 15 paragraphs, not great. If your ticket takes five paragraphs, sure, good.  But if your ticket takes five paragraphs and then later, like two days later, they have a question, and they can look back at what you wrote, and they don't have to ask you again, that's a gold star, like, that's where you win. So as an SRE, you're doing the exact same thing but for people inside your company because people are usually looking to you to be like, how do I make sure my system doesn't OOM every three days? What should I be looking at? And as an SRE, you could be like, well, you should probably add more memory, or maybe you should do this.  But if you can give an answer that's very context-specific and packaged up in a way that they can understand how it immediately applies to them and also, in a way that when they look back in five days after they've made some changes, they can be like, oh, these principles still track, then that's how you win, in my mind, as an SRE. Tech chops is great, but packaging information in a way that helps people be more reliable with the way they do things that's where you win. ADRIANA: Yeah, that makes sense. And I guess being in a support role, you have to develop a lot of empathy for your customer, which I would imagine is something that goes a long way than for an SRE role. THILINA: Yep. In support, people talk to you, and they're not being successful with your software or the thing you're supporting. So there might be some angry phone calls, you know, like, hey, let's talk, and let's make it happen. But yeah, that's correct; in SRE, you're also responding in times of things not maybe going right, and the empathy is there.  But the empathy also needs to exist where it's like, someone's trying to just get something developed and deployed. And you don't want to be the one standing between them and deployment. And when I say them, I mean engineers or application developers, even though I don't like to think of us versus them. But when someone's coming to you asking about, "Why do I need to have these autoscaling policies? Can I just ship this?" Because I can just get this done today.  The answer can't be, well, you need to be more reliable because, duh. It's more like, hey, man, you want to ship this today but what if you get paged on Friday night? That would suck. Here's how we make sure that doesn't happen. It's never adversarial. It's never I'm telling you what's right because I know. It's more like, let's make sure none of us get paged because zero page on call that's the way we want to do things.  ANA: I love that. It's actually thinking of reliability as a team sport, which I think it's not advocated for enough. THILINA: Yes. There's one more thing I was going to say about support and SRE and the parallels, which is that in support, one way to buy time is to communicate. Like, if you're working on a ticket and you don't have the answer, and they're like, well, do you have an update for me? And your update could be like, here's the update, but it might not be a solution. Same thing goes for SRE.  Like, do you know how to do this very complicated thing where my 36 sharded databases keep on OOMing at this specific time? And the answer could be like, no, but we've checked the proxies. That's not part of the problem anymore. We rule that out. We're going to look at this next. And helping people through what your thought process is, I feel like that's helped me in support and SRE, too, because an update is better than nothing. ADRIANA: Yeah, for sure. One question I had for you is how does observability factor into your role as an SRE? THILINA: Hugely. It factors in a lot. Without it, I'm kind of blind. ADRIANA: Awesome. That's what I like to hear. [laughter] THILINA: I can answer your question with an analogy and another support analogy if it helps too. So when I used to work in support for this company way back, it was around troubleshooting DNS, DNS the phonebook for the internet.   ADRIANA: Ouch. THILINA: Yeah. But let me tell you that it was a lot easier to troubleshoot tickets that were typed out than on a phone call. And the reason I say that is because when they were typed out, they would give you some data, which is obviously already better than being like, hey, I can't get to this site. It's not resolving. But also, we could ask them for a diagnostic, which would be a tool. It would do a bunch of nslookups. You could see what the results were, and then you can make informed decisions based on actual data. What I'm trying to say is working on tickets without that data was extremely tough. You're kind of going off of like a couple of common root cases or maybe some things that could happen. But you're really depending on someone to package up the information that you need, versus a diagnostic would tell you exactly what you need. And you can sift through the data and filter out, and give a better answer. Same thing for observability; without that, without knowing how your systems are interacting and working or playing together, you or specifically me, like, I'm blind. I need that data. And it's hugely important, like, very critical. Yeah, observability. [laughter] ADRIANA: Yeah, observability, rah, rah. Love it. [laughs] ANA: As you've gotten a chance to see a bit of the SRE space, what do you think people are doing wrong? What do you think folks are approaching in a way that's like, ooh, this actually might not be the best way to do it? THILINA: You can't engineer your way out of every problem, is the one takeaway I've had. And what does that mean? That is such a vague answer.  ANA: [laughs] THILINA: And my response to that is that just because you've gotten a lot of systems and processes in place to increase reliability, like, you have observability, cool. You've got an autoscaling policy, awesome. But at some point, this is going to get to the point where something is breaking, even though you've got all those systems in place. And what matters at that point is how easily can people find the information required to solve the problem? Observability is cool. It's a means to an end. It gives you some data, but that's just part of it. That needs to be tied into how you document how you troubleshoot. Just because you have observability doesn't mean that someone being woken up at 3:00 a.m. is going to know what to look for. The human element involves actually tying it all together into the human piece of, like, how do you troubleshoot? How do you integrate? This is a very vague answer.  But I think a big mistake that I've seen is people chasing after the next shiny technology thing or process. Like, I can add all these autoscaling policies. I can make sure that we never run out of resources. Yeah, but what happens when your network goes down? There are so many what-ifs. You cannot account for what it is, but you can definitely have a process or a document written that says maybe talk to this, or talk to this person. The human piece is hugely critical. Yes, that one piece. ANA: You're just walking into one of my passions, the topic of chaos engineering, of preparing for those what-ifs.  ADRIANA: [laughs] ANA: You just want these questions to be thrown at you. [laughs] THILINA: Let's do it. I'm here for it. Let's go. [laughs] ANA: What do you see as one of those best ways for a team to come together and really talk about those what-ifs and document them as you say? THILINA: If you've been reading into chaos engineering, this is probably very familiar to you. But the act of breaking your own system goes a long way. And it's not just like, hey, you've worked on this before, like, try to break it. Do your thing, sure. But I think the best part comes from what if you have a newer member on your team, and you just get them to break it and just run through that whole exercise?  ADRIANA: [laughs] THILINA: Because that's going to reveal a lot of questions. Like, oh, why do we go through two load balancers from these different paths? Well, it's because we have to factor in for this use case. Cool, but what if those people do something different? Vague answer. What I'm trying to say is look at things with new eyes and try to break something; that goes a long way. For me, recently, it's we just deployed a new service. We architected and deployed a new service, and going end to end on everything was kind of cool.  There was a point where I had to whiteboard out the headers that are exchanged between different components throughout the lifecycle. And sure, tracing will give you that; that's great. But what I'm saying is knowing the context for when those different headers and pieces come into play that only happens if you're actually going through this lifecycle with trying to do these use cases. So all that to say, try to break your stuff and write notes about it. That's a huge learning experience for me. ADRIANA: I like that advice. I actually want to go back to something that you said earlier, which is when something breaks, and you get woken up at 3:00 a.m., the answer isn't immediately in front of you. However, observability can facilitate that. So how do you use that information to help you troubleshoot? I think that's something that would be really useful to our listeners to understand. THILINA: I'll say the starting point for anything for me is always a doc. So I've always had an in case you get woken up at 3:00 a.m., click on this document and read through it. Sure, great. And that should lead me ideally to like a single pane of glass or a place where I can query for information. Once I'm there, whatever the page is, the dashboard doesn't really matter; I would like to see what my application does and just kind of run through it. And this is where the support side comes into play.  I've always seen everything from the customer's point of view, which is like DNS. I'm a customer; I look something up, I get a response. But what does my system do for my end users or the people that use it? And I would just go from, as the saying goes, from left to right and see what observability pieces kind of show me whether those things are green or not.  And this gets a little bit harder if what you're doing is asynchronous, like far-spanning. But hopefully, observability will be able to guide me to where to look and then allow me to give me the tools to drill down a bit further. But I don't think having a dashboard that accounts for every single possibility is possible. Just let me look at the different pieces.  ANA: Totally. ADRIANA: So do you have something that gives you a clue on where to start looking, or is it that, in your case, the doc is the starting point? Where do you start looking? [laughs] THILINA: The service that I kind of mentioned that we've been architecting and building out, we kind of go over the most common operations. Let's say there are three use cases, and maybe the first one is like serving requests. Alongside each common operation or use case, I usually have a dashboard or a set of things to look for associated to it. So, for example, one of the things is we serve users' requests on this endpoint. One way you can make sure that this is good is you should see the amount of requests being served be more than zero. You'll see memory being close to here, but it'll be linked to a dashboard or something to look at.  I have a weird way to explain this, and this might not make a lot of sense. But if you've ever played God of War or these old video games where you have to fight a big monster, you might get to the point where each different piece has a health percentage. You strike the person's arm, and the arm is now 77%.  I like to break down the system into multiple pieces and have each piece linked to a part in the dashboard. So even though you might not know we're serving requests or the arm does this, you can be like, let me just go check these and see what's green or not. I don't know if I answered your question. All that to say, I have a lot of different docs leading to different dashboards, and that's how I'll answer. [laughter] ANA: That makes sense. I have always kind of enjoyed at least one reference document to put everything in. Like, this is a playground to start either your alert is going to say what service, or what cloud provider, what region is actually having issues. And then go through the doc, and it's like, where can I pick up my first clue? Because at the end of the day, you're playing a detective game. You're investigating. THILINA: Yeah. Well, one of the SREs I was talking to kind of mentioned that he divides his dashboards in two ways, and we've kind of been doing it too. One is like a machine-based dashboard, which shows the defaults like CPU, memory, RAM, whatever, and that allows you to see anomalies. But then the other dashboards are either tailored around SLOs, which is tailored around what you do with your service, or tailored around common operations. Like, if you know that one specific endpoint has a lot of asynchronous calls that requires a lot of memory to be kept in state, there's going to be a RAM counter here. So that's two ways. That's a better answer to your question.  ADRIANA: Okay. Yeah, yeah, that's awesome. ANA: I mean, it's perfect because when we think about the reliability of services, specifically, it's like every service is going to have those unique reliability goals. So that is a perfect moment where service-level objectives come in because you kind of are able to turn that dashboard backwards and just give it to you quickly in a percentage of, like, an SLO is really close to customer impact. So you should actually be able to gauge from there and help your debugging go a little quicker, in a way. THILINA: Yeah, I agree. ADRIANA: On the topic of SLOs, do you have anything that you can share with regards to setting up SLOs for the first time? For example, what was it like the first time you delved into the world of SLOs? We talk a lot about SLOs in SRE, but I do know that a lot of organizations aren't necessarily super mature when it comes to SLOs. I think a lot of organizations are starting to see the value of them but haven't necessarily gotten to that place yet. So, what's your experience around that? THILINA: So I have a very interesting relationship with SL* in general. Most people, when they go into engineering, are like, oh, we built a thing. We should probably figure out what level we can say that it's performing at so we can give it to marketing and salespeople so they can sell the product. That's cool. For me, I kind of came at it from the other side where I was in support.  So SLAs are very important to me because when a customer is mad, upset, angry, frustrated, sure, they might have not-so-nice words, but also, once their use of whatever you're doing is impacted to the point of breaching an SLA, then there are consequences. Depending on how an organization operates, if you breach an SLA, something happens, not great things.  So for me, I learned how important SLAs were to start with, which probably meant that when I went into looking at SLOs and trying to create SLIs. I was very anxious about doing that exercise. I'm like, oh no, what does this mean for the person supporting? But if I was to give you a one-word answer, is that SLOs are collaborative, surrounded by two sparkly emojis. Yeah, it's collaboration. [laughter] Initially, I worked with SLAs, like, understood the impact of them being broken. I was like, okay, that's not good. And then, I started working on an engineering team, and there were already SLIs and SLOs in place, which meant for me, as a service-level objective, my job is just to ensure that those didn't get breached. So it's a guiding light for the decisions I would make. Is this thing I'm about to do in the outage going to be bad or good, hurtful or less hurtful? Is this thing I'm about to introduce into the codebase going to have second or third-order impacts that could affect the SLO? Yeah, likely, pretty likely, maybe don't do it.  But coming up with SLOs that's where you learn about SLIs. What do you want to observe and why? And when I say collaboration, you should never be doing this in a vacuum. [laughter] If the SREs are like, yeah, we're going to go away, talk about some SLOs; we're going to come back, and we're going to have some SLOs, that's bad. But for me, the first thing I had to do when I got the ticket that said, "Let's figure out the SLO strategies," I figured out who I have to talk to. That's probably like a product manager, probably people from support and figuring out what do we talk to our customers about. Having all those people in the room to talk about what you want to measure goes a long way. And then the work of building SLIs and SLOs has already been really well written about. There's a whole bunch of books, maybe from some really good writers that we all know. Building SLOs and learning about them is not hard. Figuring out what you want to learn on and measure that's the hard part, and that's where the collaboration comes in. So I guess final answer is SLOs are collaboration. ADRIANA: Awesome. I love it. ANA: It is true. You definitely need to have the right people in the room. There are so many times that people forget to bring a PM, and you need to remember that there's business logic like OKRs, like BLOs, business-level objectives. You can call them whatever you want. But at the end of the day, you're keeping everything reliable so you don't breach a service-level agreement and have to pay your customers money for not upholding the standards of service that you need it to. So definitely think about the business. THILINA: Yeah. And I'll give you a good example of that. When I was a systems engineer, the thing we were working on was considered a critical tier zero service. So, for me, that meant if I got paged, I had to respond within three to five minutes and resolve it immediately because if I didn't, some small country might lose a big portion of X service for 30 minutes, which is not good. So for me, I was always like, pager is going off. SLOs are being impacted, must impact bad; go do this now. And then afterwards, I worked in a different company as an SRE where it's like, there was an alert going off for P1-related thing for like 30-40 minutes, and people were not super stressed out. I was like, why is everyone not stressed out? This is terrible. This is crazy.  And the PM was like, yeah, but the assessed value of this is not very high. It can wait, like, that's okay. That's why our SLO is actually lower. It doesn't have to be super high. But if it was just me in a corner making SLOs, I'd be like, nope, 99.9999% very important; that's how we do it. Reliability: that's my job. Not true; you're helping the rest of the org. So having more people in the room is very helpful. ANA: That is true. You're almost going to set yourself up for failure if you don't talk to the right stakeholders. You're going to have a 99.99% SLO and your business need actually may have been just like 92%, and then you over-engineer, which then costs your business a lot more money. [laughs] THILINA: And then you train your users for a higher level of stability or reliability than you're actually supposed to be doing, and then people have a disconnect in expectations. ADRIANA: Absolutely. You mentioned one thing that I want to go back to with regards to SLOs, which is I think you implied that having your SLOs also keeps you accountable in terms of when making changes to the system. But then I guess the other side of the SLOs is also that they are a moving target of sorts because you're always iterating on them. In your experience, can you talk a little bit about iterating on SLOs? THILINA: When I think iterating SLOs, that obviously means tweaking the number to be whatever the business deems it to be. And so when I say that you're making decisions based on the SLO, it doesn't mean if I do this now, is this going to break SLO? I'm thinking more that the SLO is to help you do some prioritization of the changes you're about to make on a system.  So if you know that something is considered an SLO that's ranked higher on the list of SLOs, if you do ranking, and something isn't, that's the thing that's prioritized. Maybe you're making changes that affect it; you should have a bit more due diligence. Versus something, like, where do you want to concentrate your effort? And that helps you prioritize. ADRIANA: Yeah, that makes sense. What about iterating on SLOs? Have you ever been in a situation where you're like, okay, we came up with this SLO, and then you have an incident, and you're like, er, that didn't quite cover it? [laughs] THILINA: I have the opposite story, actually. ADRIANA: Oh. THILINA: Which is at one of my organizations, every time an SLO was breached, that usually happened due to an incident. And that meant that there would be a retro or a postmortem where everyone would get together and do the ritual and the ceremony of, like, what went wrong, and why was this bad? What was the impact? In that case, we found ourselves in a string of repeated postmortems. We were like, oh, why are we here? Oh, something went down. What was the impact? Not that bad. Then why did we get paged? Oh, because our SLO is set to this.  And then that led to the question, should it be set to this? No. Okay, well, maybe let's tweak it down. Does that sound good? And it was like, yeah, sounds good. It was like, all right, cool, let's do it. And then we didn't get paged as much. Yeah, things still happen, but that was tweaking it in a way that's making it less sensitive because we found that it wasn't as big of a priority. So kind of the opposite story of what you hear about. ADRIANA: Yeah, that's actually such a great thing to bring up.  THILINA: That story really depends on SLOs being part of the process as opposed to just a thing you chase for. Having SLOs on its own doesn't do much for you other than help you tailor your dashboards. But having SLOs be part of your incidents, your retros, your postmortems, your reliability reviews, other people have to care about your SLO than just your sales folks and you as the SRE.  ANA: Yes, seriously. It goes back to reliability is a team sport, like, collaboration. THILINA: 100%, yeah. ADRIANA: I love the story of revisiting the SLO and realizing, like, oh, it was too sensitive because that is just as important because if you're getting over-paged, you're getting alert fatigue, basically. That's not good for anyone. You give your team PTSD, and then they don't start taking the alerts as seriously, or they jump at every little thing. And that's not healthy for your team and, therefore, not healthy for your system because if you have an unhealthy team, well, it'll lead to an unhealthy system.   ANA: Burnout. It's called burnout. [laughs] And your team leaves, or word gets out of this is a firefighting organization, don't go work there, which definitely happens a lot. On that topic of iterating on SLOs, I always have felt that SLOs make it a lot easier to adjust those pagers to those alerts to actually not get pager fatigue versus having them for every single service based on dashboards and going that way. It's harder to tell when it actually matters to the customer when you're looking at that metric in a dashboard versus this is the SLO within my dashboard, within my service configuration. THILINA: Yeah. Actually, one golden tidbit that I would like to pass along for SLOs, and this is like, if there's one thing I learned from this experience, is that if you're starting to tweak an SLO, do a look-back analysis. What I mean by that is the system we're setting up is going to be taking over for something else. And we noticed that the SLO for the old system was set lower, and the one we wanted to set up was higher.  And the question we wanted to ask ourselves before we implemented this was based on this higher SLO that we want. If we had applied this looking back to this existing system, how much more would we have been paged, or what would the impact have been to the team? So doing a look-back if you're going to tweak up for sure would help answer that question or guide you more. Because if you say I'm going to increase something from 97 to 99, you're like, oh, great. That's awesome. That's great. Reliability: awesome.  But if we say we're going to increase from 97 to 99, but if we look back in the last two months, that means a team would have been paged 36 times as opposed to 20, oh, that's a little concerning. How important is reliability? Is 2% really worth 16 pagers? Like, what's the impact here? So look back, analyze against previous data. That's a huge thing I learned from my first experience setting up SLOs. ANA: That's an amazing golden nugget that a lot of folks don't always take into account, like, consider what could have happened as history stands but also look into the future and say, well, that means that we are only allowed to be down these minutes. So we have to be a lot faster about triaging incidents and things like that, making sure that you also think about that downtime part. Your error budget, I think, is the word I'm actually trying to think of in my brain [laughs] that I couldn't get to.  But with your golden nugget, it made me think of the question of, like, as you're sharing your experiences of going through this, do you have a platform where you share with folks some of these golden nuggets in a longer format? THILINA: Yeah, I mean, I have my blog that I've been really making an effort to update more; it's tratnayake.dev. One thing that's important to know about my blog posts is it's not meant to be...like; it's definitely more like a lab notebook that I keep with me as I learn things. So, for example, what I'm going to be writing about is me talking about a change I made in Envoy that's going to go into detail about what is Envoy if someone was reading it for the very first time. So if you're into blog posts and learning things about stuff that you might already know because we get really elementary, you should totally check it out. ADRIANA: We will link to it in the show notes. Thank you. ANA: That is pretty neat. And as folks eventually get to hear our season finale, you'll get to hear why sharing your learnings as you go really do matter. It just makes the industry a lot better. [laughs] THILINA: Oh, sharing learning is huge. Like, if there's advice I could give to a junior engineer, just get better at explaining what you do and what you're thinking. That will serve you more than learning regex. That's a spicy take. ADRIANA: Yeah, it's so true. Well, I found with personal experience, it's one thing to solve it for yourself. It's another to solve it for yourself and then write about it, and then realize that there were some gaps in your brain in understanding because now you're trying to explain it so someone else can follow the instructions. And so you got to do a little extra research, which is good because you learn some more stuff along the way. THILINA: 100%. Writing more things down makes your team better and makes your team faster. One big thing I took away from support is if something's not clear to a customer, there should be a doc for it, so immediately write a doc: cool. And then, in the future, you can always just send back that doc as opposed to having to answer that question. But for me, I keep notes on everything I do, like process-related.  So I had to deploy a release for a thing two months ago, and I wrote about it. And I was like, well, it's just for me but whatever. And then this was supposed to be a rare operation, but this became less rare, and we did more releases, and some teammates had to do it. And I just gave them my doc. And as opposed to me having to walk them through for two hours just like I had to learn it, they were able to just take the doc, do what they needed to do, and ask me when they run into questions. And that made the team faster. So write things down. You never know when it's going to help you. I'm a huge proponent of writing things down.  ADRIANA: Yeah. And then now you're no longer a bottleneck.  THILINA: Yes. ANA: Your future self or someone else in the community will thank you for it. ADRIANA: Yes, yes. And I would add, add lots of details because my biggest pet peeve, and I've said this multiple times, is when people will write a technical blog post and either assume that I know what the hell they're talking about, or they're just tired of writing. So they'll leave out details, and I'm like, but I don't know what you're talking about, dude. [laughs] THILINA: Oh, yeah, err on the side of over-communication, again, just provide more context. It doesn't hurt. ADRIANA: Yes. ANA: Add all the damn links. Let the user decide if they want to follow the path to them. [laughs] ADRIANA: Yeah, exactly as opposed to...do you remember those math books in high school that will leave the proof up to the individual to prove, and it's like, nooo, I don't know how to prove the fucking proof. [laughter] Oh. ANA: I freaking hated that part of math, by the way. ADRIANA: Oh my God, me too. It's like, no, I will not. Don't leave it up to the individual because the individual has no idea. [laughs] THILINA: Correct. ANA: So you're teaching me things. Why are you not teaching me what I paid you to do? [laughs] ADRIANA: Yeah, exactly. So yes, over-communicate and down. ANA: As we're getting to wrap up the episode, I wanted to ask a little bit more about what your life as an SRE is. What is on-call at Lightstep like? THILINA: On-call is really nice here. It's very, very nice. And I say that because I've been part of organizations where P1 outages would be a very common thing multiple times a week.  ADRIANA: Ouch. THILINA: That is not the case here. But I think more than that, on-call is very nice because we ruthlessly prioritize silencing the things that go bump in the night. And what I mean by that is not just like, oh yeah, just put it in another room; we're not going to hear it anymore. It's more like, oh, why did this go bump in the night? Oh, because this happened? Cool. We're working on that immediately now to make sure that the next person that's going to be on-call does not get woken up by that.  That ruthless prioritization of ensuring that we reduce the impact to on-calls livelihood is huge. Also, the fact that I really enjoy that we will write down the things that we see that could be improved or could be going wrong, and then people will read it and actually take the time to work on it. Like, that's really cool.  ADRIANA: That's awesome.  THILINA: We've talked about burnout. I think we mentioned burnout earlier. And burnout can come from many different ways; one of them could be pages and being fatigued from alerts. Burnout can come from doing too much too often. But burnout can also come from trying to write things down and raise awareness to a problem and not being heard. That can lead to a lot of burnout. And I'm very happy that whenever we have something to bring up or we write down, we'll read together and we'll discuss, and that goes a long way. ANA: That's awesome. And when you're on-call, what are some ways that you make sure to take care of yourself? THILINA: When I'm on-call if I'm going to be away from the pager, I'll always be like, hey, I got to go do this thing for an hour. Is someone able to cover me or just keep an eye on things? Again, erring on the side of over-communication really helps. But also scented candles are really nice. And the reason I bring that up is incident response at 3:00 a.m. with a scented candle game changer. Game changer. [laughter] You're getting ready for that conference bridge, and you know C-suite people are about to join; you got a scented candle. You got some vanilla tapioca in the background. Everything's fine. We're good. ADRIANA: Wow. ANA: [laughs] Do you have a favorite brand of candle? I think I need a link here. [laughs] THILINA: No, I just have smells that I would recommend. Definitely, holiday vanilla is one of my favorite smells. I don't know if that even transcends to different places. But yeah, vanilla is a great smell. ANA: It's a very soothing one, so I totally get that. And on that realm of questions, what is the best way your team or manager has supported you while you've been in an incident? THILINA: I was going to bring up that it's super great when managers bring you pizza, but that's like an old thing that people hear about outages and incidents. ANA: All the time. [laughs] THILINA: Yeah, all the time, yeah. [laughs] My manager has supported me; previous managers have supported me during an incident. It's kind of helping me run interference if I'm leading an incident. So like, if I'm on a call and I'm doing things, just helping out by being like, "Oh, this question was answered here," or like, "Check over here," kind of being like a secondary gopher or runner.  But more importantly, during an incident, if my manager DMs me and is like, "Hey, I see you've got this. Is there anything you need right now? Or is anything you need later? Just shoot me a text or just even give me a call, and I can help you out." Just letting me know they're there is very helpful. It goes a long way. ANA: Just show up. [laughs]  THILINA: Yeah, just show up, yeah.  ADRIANA: That makes a huge difference because there's nothing more annoying than thinking that you've been abandoned by your manager, so... ANA: Most definitely. So making sure to over-communicate and put people first. We are working on systems. We want to keep them reliable. By the end of the day, there are people behind those systems that are trying to keep them up and customers also. THILINA: Yeah, and I think that over-communication piece how that applies to now is that when we were in the office, people could see you. And for me, I'm very emotional, so people can see when I'm not feeling great. But when you're behind a screen, you can't really do that. So I've made use of being like, hey, can I talk to you, or can I vent to you about this for like five minutes? Or, I'm very frustrated about this.  All my managers have been very good about that, just being like, "Yeah, let's just jump on a call. Let's talk about it," just being there. Showing up is good, but being there and just being available to take those really quick calls goes a long way, especially if you're remote. I highly recommend that. ADRIANA: Yes.  ANA: I love that advice for managers. Just being there really does help. It's true. I've had a fair share of managers or managers where I'm like -- [laughs] ADRIANA: Yeah, managers take note. ANA: Well, I love that as our last golden nugget for today. Thank you so much T, for joining us in today's podcast. Don't forget to subscribe and give us a shout-out on all social media via @oncallmemaybe. Be sure to check out our show notes on oncallmemaybe.com for additional resources and to connect with us and our guests on social media. For On-Call Me Maybe, we're your hosts, Ana Margarita Medina... ADRIANA: And Adriana Villela signing off with...  THILINA: Peace, love, and code.

About the guest: Marylia is a Toronto-based Engineering Manager and Developer at Cockroach Labs, working on Cluster Observability. Before that, Marylia was a full-stack developer at IBM, working on internal Observability tools for DB2 products. Find our guest on: Twitter LinkedIn GitHub Find us on: On-Call Me Maybe Podcast Twitter On-Call Me Maybe Podcast LinkedIn Page On-Call Me Maybe Podcast Mastodon On-Call Me Maybe Podcast Instagram On-Call Me Maybe TikTok On Call Me Maybe Podcast YouTube Channel Adriana’s Twitter Adriana’s Mastodon Adriana’s LinkedIn Adriana’s Instagram Ana’s Twitter Ana's Mastodon Ana’s LinkedIn Ana's Instagram Show Links: Cockroach Labs Chocolate Velvetiser UX - User Experience IBM Cockroach DB Console to Observe and Troubleshoot SQL KubeHuddle Toronto’s BeaverTail History Flex Fridays Additional Links: Charity Major’s The Engineer/Manager Pendulum Will Larson’s Staff Engineering Blog & Book OpenTelemetry Collector alongside CockroachDB Brazilian Can Opener (Abridor de Latas) Transcript: ADRIANA: Hey, y'all. Welcome to On-Call Me Maybe, the podcast about DevOps, SRE, observability principles, on-call, and everything in between. I am your host, Adriana Villela. And with me, I've got my awesome co-host... ANA: Ana Margarita Medina. ADRIANA: And today, we are speaking with Marylia Gutierrez, who is an Engineering Manager at CockroachDB. Welcome, Marylia.  MARYLIA: Hello.  ANA: Welcome. MARYLIA: Thank you.  ADRIANA: First things first, where are you calling from today? MARYLIA: So I'm calling from Toronto, Canada. I'm actually from Brazil, São Paulo, but I've been living in Canada for the past almost nine years now. ADRIANA: Awesome. Awesome. I'm also in Toronto, Canada. And Ana, you are recently back from SREcon, right? ANA: Yeah, I actually just got back from Santa Clara over to the Marin, California area. And [laughs] it's really nice and sunny today. But now I'm actually outnumbered by Toronto folks, so this is pretty neat. [laughter] ADRIANA: Toronto folks and Brazilian folks.  [laughter]  MARYLIA: Yes.  ANA: True, true, true. [laughter]  ADRIANA: This is an all-Latina cruise, so yes! [laughter] ANA: Latinas in tech, we're here, and we're proud. [laughter] MARYLIA: Yes. ADRIANA: Let's start off our questions round with what are you drinking today, Marylia?  MARYLIA: So I have hot chocolate with me. Hot chocolate is always my go-to drink. [laughs] ADRIANA: Very nice. And you were saying that you are quite the hot chocolate aficionado, right? [laughs] MARYLIA: Yeah. So usually, people have at home their set up for coffee with a grinder and everything. For me, it is the hot chocolate. So I have a velvetiser that actually can melt the chocolate, mix it with the milk, and I put the right temperature. And even the glass that I'm using has the shape to remind of cocoa, like the stripes of it. [laughs] So yeah, it's all chocolate themed. [laughs] ADRIANA: That is awesome. ANA: That's so cool. Is there anything special about the flavor profile for today's hot chocolate drink? MARYLIA: No, not really. Actually, I was having one that I just went to, like, a random store, and I bought it, and it was delicious. And it ended this week. But now I need to go back and actually pay attention to the name and [laughs] where did I buy it. [laughs] ADRIANA: Oh, that's the worst. You find something awesome, and you can't remember what it is. MARYLIA: Yeah. [laughter]  ADRIANA: What are you drinking, Ana? ANA: I am drinking a strawberry acai refresher with a little bit of sparkling water. I had the Starbucks drink, and I was like, I need it to be fizzy, so I added a passion fruit LaCroix. And it's actually really refreshing because it's actually really hot out here today in my office, so I'm enjoying it. What about you, Adriana? ADRIANA: I'm drinking some green tea. I just brewed myself a cup just before the podcast today so that I'd have something more interesting to drink. Pretty soon, we'll be into bubble tea season for podcast recording, so I'm excited to bring that. [laughter]  ANA: I have to get on that, but I'm very excited to learn. [laughs]  ADRIANA: Cool. Well, let's get started with the serious questions, although we never take ourselves super seriously here anyway. [laughter] So, Marylia, how did you get started in technology? Why don't you share that story with us? MARYLIA: Growing up, I actually never really knew what I wanted to do. And then, at some point, I got the computer at home, and I thought, oh, this looks cool. But all the applications, everything was gray and square, and stuff, so I was like, why everything has to be this way? Can things be easier to use? So I was always getting frustrated with things. So that caught my attention, and I was like, I want to work on something but make it useful for the user. So I wanted it to be easy to use. So that started to bring me a little to that side.  And then, I went to university, and I graduated in computer engineering. I also did my master's. So for the master's, I focused a little more on the UX side, so that was an area that I was very interested in from the beginning. And then just going over internships, jobs [laughs], and things just kept evolving on that. ADRIANA: That was super cool. When we were chatting earlier, right now your current role you're an engineering manager at CockroachDB. It sounds like early in your career UI UX was kind of your thing. But now the type of work that you're doing is very different from that. So what can you say about that? MARYLIA: My first full-time job was a mobile developer [laughs] so I even continued on the UX side that I was working on. And then, after that, I actually went to IBM. And on that one, I started looking a little more on the observability side. So I was working on an internal tool. And again, we had a bunch of data collecting about usages, things that people were doing. But again, it was hard for people to understand what all those metrics means, what all that information meant.  So I started creating visualizations on top of that, like dashboards and tools and things that make it easy. Like, I'm about to call a customer now. I need to know everything we know about that customer. So you could filter on just the customer name, and it will have all the information about the cloud storage that they have, how much they're paying, how many tickets they had, and how much time their cloud servers were taking to be created, and all of that. So I started to really enjoy that part. And so I was there for seven years at IBM.  And then two years ago, I was looking for a change, and then this is why I moved to Cockroach Labs. And then I wanted to continue a little on that area. But I think the important part for me was to continue being a full-stack because I always liked being a full-stack. And when I was choosing even the team, I say, okay, I want one that I can continue being a full-stack. And they didn't have a team for SQL observability at the time, which is the team that I manage. So the team was being created.  So I saw that opportunity for me to continue working on the two sides on an area that I was interested in. And then I can continue going back to my initial desire of, okay, now you're using a database. You have a problem. How do you want to fix it? Again, I want to make it easier for the user to know what is the problem with the database. I want to make an easy way for them to fix the problems by themselves, or we can already fix for them. So in the background, I still continue with the [laughs]. I want to improve things for the user.  So when I joined there, I joined as a developer. But very early on, I was helping with a lot of other things, and I just actually became the manager for the team. And actually, just this week, we are migrating two of the observability teams, the SQL observability and [inaudible 7:25] observability, creating just one for cluster observability. Then I actually I'm the manager now. ANA: That's super exciting. And I think a lot of folks don't realize that when they start in the industry, whether it's front end or back end, you could eventually bring in all your skill sets together, like you said, bringing in that front-end UX mindset to observability. Like, yes, we all care about the information out there and the context this provides. But how is it that we can make sure that the consumer and the user is able to take that and build proper mental models or even just have the right information at the right time? MARYLIA: Yeah, exactly, because I can give you here all this 20,000 files of logs. [laughter] You're like, okay, what does that mean? [laughter] So I'm there to, like, put in, okay, this is in your face. This is the thing. Fix this. This is what you should care about. ANA: I think it's that, like, make sure you highlight the information that you should be caring about and defining what that is.  MARYLIA: Yes, yeah, exactly. ADRIANA: Yeah. It's kind of funny how we're so used to working with logs. I was part of a webinar a few weeks ago, and one of the questions that was asked was like, what was your first observability thing, observability signal that you were introduced to? And most people basically, their signal was logs because, as software engineers, that is the thing that we use to try to make sense out of our systems.  And it's one of those love-hate relationships where it's like, yes, you give me the information I need, but, oh my god, [laughter] this was like finding a needle in a haystack. And I love that observability kind of takes it to that next level where now it's like a trace-first approach where logs still play an important part. And now you get a little more context into what's actually happening with relation to the big picture, which I absolutely love. MARYLIA: Yeah. Even giving the example on CockroachDB, so we collect a lot of metrics. The user they have access. They can go and look by themselves. But my team we're also responsible for the console. So we are creating pages on top of that information. So we have one page that is like SQL activity. You can again see everything there that you want to dig into and decide. And then we have a page that is insights that will tell you, oh, these are the things that you should be paying attention. These have a high contention. This is having this problem or that problem. Then they can focus on the big ones first, like, okay, these are my main issues; fix them. And then they can continue looking at whatever they need. ADRIANA: Are you finding that your users are getting a lot of value out of that kind of thing now? Being in a position where you got to see this thing built from scratch, are you seeing your users going [gasps]? [laughter] MARYLIA: Yeah. So as soon as we created this, I had somebody using it send us a message, "Usually, it takes me ten days to do the thing that now I was able to do in 1 hour," because one of the features that we have is index recommendation. And then they were like, "Oh, I had to actually find what statement was in a bad shape, which one I should be creating an index for, but now it shows right away. This is the one you should be creating."  At the same time, we show this index you created all of those, and you never actually use it. You should be dropping those. And it is right there on the console and very easy to find. So they're like, "Oh yes, please." [laughter] ANA: Getting a chance to win a customer or a user like that of I just made your very complex job just a little bit easier, and I'm helping you find that context a lot faster is very satisfying. That was actually going to be my question, like, how is it that your team goes about getting that user feedback from the users on the observability side of stuff?  MARYLIA: Yeah, because we also have a PM on the team that also goes to the customers just to see how they are liking it, getting any feedback. So we also get the context from that side. But I think even somebody was asking on Twitter or something, and then I explained how the insights part works with the index. And they're like, "Oh, you are making me lose my job as a DBA. This is doing my job." I was like, "No, I'm just making your job easier because now [laughs] you don't have to spend the time." [laughs] ADRIANA: Like, I spent the first chunk of my career heavy into using Oracle. So performance tuning Oracle, I mean, like any database, really is a heinous, heinous, heinous exercise because you're trying to tune SQL queries and database tables, and the database itself and not knowing where to start is kind of a nightmare. And I think after a while, you develop a certain intuition around it. But isn't it nice now that we actually have something [laughter] here to guide around these things? ANA: It's like, database is already hard. And if we just make it easier to know the status of them, to know how they're doing, like, we're starting to solve that problem of just bringing in, like, lowering that complexity, I guess. But still, understanding that that's an abstraction layer and the system is still very complex. [laughter] MARYLIA: I was going to say that when you talk about databases, people are like, "Ah, no, I don't want to have to deal with the database."  ADRIANA: [laughs] MARYLIA: So, ideally, you create your application. You have your database there and you never have to think about it. So that is the ideal scenario. [laughs] So if you have to interact with this, let's make them as smooth as possible that you can use. ANA: Most definitely. You're currently an engineering manager, but you worked as an individual contributor for a bit. How is it that you made the transition, and how do you feel about it? [laughs]  MARYLIA: It is funny because I always said, "No, I'm never going to be a manager." [laughter] A lot of ICs are like, "No, not for me. I don't want it; go away." [laughter] So I was an IC for almost ten years. And then when I joined Cockroach Labs, I was thinking like, okay, should I be or not? When my manager talked to me about it, he was like, "Oh, you are doing a lot of things that a manager does. Do you have an interest in it?" So he was like, "Spend some time talking with other managers."  So I talked with a lot of other managers at the company, a few that had already left. And I was like, you know what? I think I want to try. And the good thing is that Cockroach Labs also has some trainings specific for people that are just becoming a manager. So I knew that I was going to have the support that I needed. It was not something that they would spring up like, you know, everything. Go do it. So I will have people backing me up during this development phase. And the manager said, "Worst case is in one year, if you decide you hate it, go back to being an IC if that happens." [laughs]  I ended up really liking it because, as a manager, I really like being a full-stack because I like to work a little on the back end and then switch to the front end. I like switching a lot. And I was like, okay, now I have a third thing that I can switch with. [laughter] So during the week, I have a rotation of each of the three [laughs] areas. And it's been really nice also helping the team, seeing other people grow. And the other topic was I don't see a lot of women being a manager, and that was also a big point for me. Then I was like, okay, if we have more... [laughter]  ANA: Yes.  MARYLIA: Even for example, I was doing some events and we could talk with people that want to become interns and stuff like that. And then I went to one, and I was supposed to stay there. I think we were dividing on slots, and I had half an hour for me or something like that. And then one woman came, and she's like, "Oh, are you the manager?" I was like, "Oh yeah." She's like, "Wait, are you a manager?" I was like, "Yes." [laughter] "Oh, but you still develop." I was like, "Yes. A lot of times, I'm still doing a lot of demos." [laughter] She's like, "Okay," and then she disappears. [laughter] I was like, okay. And then a couple of minutes later, she came back with like ten other women. [laughs] Like, "It's her. It's her." [laughter] I was like, "Hey, what is happening?" [laughs] And one of them was like, "Okay, I never saw someone like you before." [laughs] I was even a little shocked with all the reactions. And they're like, "Oh, you're my role model." And I was like, "You don't even know me." [laughter] I think that was a big also factor of even if it was something that I was afraid of, just me being there gives the right message to others that want to become. ADRIANA: Yeah, totally. I do find seeing peers at my level who look like me, whether it's as a Latino or as a woman, I think you kind of take it for granted that it's not something that you necessarily consciously think about, at least not me. But subconsciously, I think we take a lot of comfort in seeing people who look like us in similar positions. And it makes us more comfortable being able to reach out to people like us in similar positions to learn from each other or to get advice, whatever. So it's really nice that you're able to do that for people who want to do this. [laughter] MARYLIA: Yeah. And I got very lucky because I joke that when I was under SQL observability, I had one meeting with all the managers for the SQL side, and I had one with other managers from the observability side. And both groups, all the managers were women, so I was like, yes! [laughter] ANA: That's amazing. MARYLIA: So on the SQL side, it was me and two others, and then on the observability, it's me and one other. So I was like, yeah, all the peers, yeah. [laughter] ANA: That's pretty badass. [laughter] And, like you mentioned earlier, that aspect of belonging really does make a difference. And when you do have those peers that accompany, it also just gives you such a way better fuzzy feeling of like, we're climbing this corporate ladder and saying, fuck you, tech industry. We belong here too. [laughter] We might not be a high percentage. [laughter] I definitely wanted to come back to the point that you mentioned on your workplace, encouraging you to be a manager but also letting you know if it doesn't work out, you can go back to IC work. That's super solid. And a lot more workplaces need to have those arrangements with their team of have fun, experiment in the organization, and we will still hold an IC spot for you. It's like, this whole people management and playing politics is not for you. [laughter] MARYLIA: Yeah, because it's always going to happen. People work for a while and realize, oh, it was not really for me. It's not a matter of competence or anything. It's just like it's not the right fit. So you don't have to feel like, oh, that means I have to leave my company now because it shouldn't have to be this way. It should be like, oh, I was working fine as an IC, so let me just go back to being an IC. That is where I'm happy. And then the company is going to be happy too because then you're doing a better job on the position that you're a better fit for. ADRIANA: Yeah, I totally agree. I think it's very comforting to have that option. Like, when I was at Tucows, and I joined as a manager, I was also kind of like, I don't know if I want to do this. And they're like, "Just try it for a bit and see how it goes." And it's very comforting to know that.  And then the other thing that I want to mention is that it's cool, too, that you're not committed to either IC or manager for the rest of your career. You can jump around as many times as you want. I've seen tons of people in the industry do that. Or you can find one that, you know, you love one of the two, great, and stick with that, and that's okay too. MARYLIA: Yeah.  ANA: I think in the last three, four years, we've been having a lot more of those conversations because I think a lot of people felt that in order to grow their career, they had to be managers. But we're also just saying like, hey, there are various paths. You can take one path for a while and then go back to another. That discussion of, like, it's okay to pursue staff engineer, distinguished engineer, and then go into management or from senior and jump to management and then eventually come back to staff engineering. MARYLIA: Yeah. A lot of people are like, oh, I have to become...it's the next step. I never saw it this way. And I don't think a lot of people see it this way. Continue doing whatever is good for you; that is what you should be doing. [chuckles] ANA: That's definitely the best way to see it. Do things you're passionate about. They bring you joy.  ADRIANA: In order to make people feel like they're not losing out by not becoming a manager, is this whole idea of actually getting compensated fairly as an IC. You do complex brain-draining work. And it's different from a manager role, but you should get compensated just as well for it. It's nice to see that the industry is finally recognizing that management is not, you know, it's a career track, and things diverge. You can either go as high as you can in the IC track or as high as you want to in the management track. And both are okay, and you're still going to get paid well. And I think seeing organizations compensate people fairly on both tracks is really encouraging. MARYLIA: Yeah, it's not like one position is better than the other. It's that one position is better than the other for you, [laughs] not being better. ANA: And it also creates that space where you say we value technical skills; we value soft skills. And it's not like the technical skills are what matters the most or the soft side is what matters the most. It's kind of balancing it out. So you mentioned that you're also from Brazil, like Adriana. And you also live in Toronto. [laughter] What was it like for you to move to a new country? [laughs] MARYLIA: Of course, when you move...I got super nervous because all my family and friends were in Brazil. And so I knew pretty much my partner, that also came to Canada also from Brazil. [laughs] And I knew another couple from here, and that was it. And at the same time, getting a new job, so that was when I started working for IBM. So I always got nervous, like, okay, what is happening? So many changes. Is that going to work?  But it was a very easy change in the sense that it was a lot safer here, just the quality of life. I was just feeling a lot of things that just in general feel more calm here and things like that, more safe. And so I think that was a good change. It is easy to get used to something good [laughs], and I think Toronto is a very good city. So, of course, you still miss family and friends, so we try to go and visit them. [laughs] In general, it was a very easy change. ANA: I get to go to Toronto for the first time in a few months for KubeHuddle, where Adriana and I will be speaking. MARYLIA: Nice. ANA: And I'm very excited. [laughter] ADRIANA: Yeah, maybe we can actually meet in person. ANA: It would be nice to start having On-Call Me Maybe photos with guests in the cities that they're from. ADRIANA: Oh my God, yes. ANA: Like, that's going to be fun. [laughs] MARYLIA: Yeah, you have to visit all the cities.  ANA: On-Call Me Maybe meetups are going to be happening, apparently. [laughter] ADRIANA: Now, speaking of Toronto, one of the things that you have to try when you're here is something called a beaver tail. It's basically a flat doughnut that is shaped like a beaver's tail, and it's covered with cinnamon and sugar or Oreo or apples or whatever. It's delicious. [laughs] ANA: I wonder if this is similar to what they call funnel cakes here. ADRIANA: I mean, we have funnel cakes here too. I think funnel cakes are definitely a lot more greasy. ANA: Okay. I will be very down for this beaver's tail, then. [laughs] I wanted to touch on the part that we just had a discussion on moving to a new country. Like, is there any advice you have for folks that are doing that transition or that are on the fence? Like, for you, was it thinking about moving to a place where you already had a community, or you knew there was a Brazilian community? Or you just knew there was going to be an actual building for your office that you can meet people. [laughs] MARYLIA: So it's going to vary a lot for people. So I think the first thing is that when you hear a lot of the "Canada is looking for Brazilians," first of all, no, that's not happening. [laughter] I get a lot of messages like, "Oh yeah, they're looking specifically for Brazilians." I'm like, "No, you have to apply normally as everybody else." And it's not an easy process for emigration. There are a lot of things to do.  And then, again, you have to consider where you're moving. So try first going to the place that you plan to go just as a tourist because a lot of times you have that image of this place that you see in movies. You're seeing stuff, and then you go to the place, and you actually hate it. [laughter]  So I got lucky enough because I knew people that were here before. So my partner actually came before me, so he came a year before I came. I already knew how things were working. So when I moved here, I had a place to stay. [laughs] It was not that I had to actually look for a place because imagine if you just move, you have to look for something. It's also another issue that you have to think about.  And again, thinking if you're not coming in with a job, you also have to consider that you might take time to actually be able to get a job. So you need to have the savings enough to survive a few months with rent, food, commutes, and all this extra cost that you're going to have. So it's not going to be super easy if you don't have a support system.  So it's always easy if you go to someplace that you know somebody that can give you some guidance. Or even like when we came here, they were like two types of buses, one you have to show the pass when you get in, the other you don't. Silly things like that, they're like, you have no idea. Or even the first time, I had to sign a check here, and it was very different from the one from Brazil, and I filled out everything wrong. ADRIANA: [laughs] Oh no. MARYLIA: And they were like, "What are you doing?" I was like, "That is the order." [laughter] Just having somebody to help out with, even with the silly things, is very important, I think. ADRIANA: That's interesting. It's easy to forget the little nuances between countries. I came to Canada when I was 10, so I was riding on the coattails of my parents. [laughter] So I didn't have to deal with any of that stuff. So I essentially grew up on the system here. But yeah, when you're coming from a completely different system, or even I would imagine folks moving from Canada to the U.S., it's a bit of a culture shock too.  It's so trippy seeing some of the differences, even some of the terminology or the spelling, [laughter] you know, things that you take for granted in Canada or in the U.S. If you're going in the other direction, there are always the little cultural nuances that you forget until you're immersed in that culture. MARYLIA: When I moved, I pretty much brought two bags of clothes, and that's it. I didn't bring any kitchen supplies or anything like that. So we had to buy the things. And even silly things like can openers, I don't like the ones from here. I was like; I hate this thing. [laughter] It is huge and keeps getting rusty. So when friends came to visit, and they're like, "Do you need something?" I was like, "Yeah, get me a can opener from Brazil." [laughter] So it's always like those little things like, oh no, I need that specific tool that I use that I cannot find here. [laughs] ANA: I relate to that so much. I grew up in Costa Rica and now living in California. There are still appliances or bowls that I have from back home because, one, it's nostalgic. But there's a part where it's just like, the quality is going to be better. But it is just more like the user experience is better. [laughter] And I'm just like, damn you, America. Like, why don't you also sell this version too? [laughter] ADRIANA: One of the biggest surprises that my parents were always bitching about when we first moved here was they were using electric stoves here. And they were like, "What the hell is this crap?" [laughs] And so they're like, "Why is it taking forever to boil water?" [laughter] MARYLIA: I thought electric showers were common everywhere. And then after I moved here, I realized, like, no, it is not common in other places electric showers, and actually, it's a Brazilian invention. I was like, oh, okay. So even at work, I was explaining like, "Okay, but what is an electric shower?" I was like, "Oh, just think about a wire that you put electricity, and you pass the water to make it hot. It's very safe, people; come on." [laughter] ADRIANA: I remember as a kid having to turn on like a pilot light in the bathroom to heat the water.  MARYLIA: Yes. ADRIANA: That's my memory from when I was a kid. [laughs] ANA: We had to do that in Nicaragua, I think, where it was like you had to power off something else and then run the water for a good like 5-10 minutes, wait for it to get hot, and then you can jump in. [laughter] ADRIANA: The things that you take for granted. ANA: So we have two more questions as we're wrapping up. If you had a few more hours in your day, what would you dig into tech-wise? MARYLIA: It would be more continuing on the UX side. I also like the accessibility area, that I haven't been able to pay too much attention. So that is an area that I would like to study more. ANA: Nice. That's exciting. Yeah, there's definitely never enough hours, and technology moves so fast that you're like, I want to do all the things. MARYLIA: Yes.  ANA: But, wait, I can't. [laughter] ADRIANA: Yeah, it can be pretty overwhelming. Sometimes you just have to be like, okay, I'm going to focus on one thing. I'm just going to chill with that, and that's it. That's all I have mental space for. MARYLIA: [laughs] So, in Cockroach Labs, we have what is called Flex Friday, so it's the day that we don't have any meetings. And then, you can work on something or continue on your regular day-to-day tasks but also a day that you can focus on studying something. So I have in the browser I create a tab for Flex Friday like reads and stuff, and I keep putting them. But I have so many that when Friday comes, I don't know which one goes first. And I end up not looking at them. [laughs] ANA: I love that idea of putting all the things you want to read during the week on a tab on Google Chrome and then coming to it at a dedicated time and being like, this is my learning time. I think you just gave me an aha moment on how to deal with all the things I want to read. [laughter] ADRIANA: Yeah, that's a really good idea. Good hot tip.  MARYLIA: [laughs] ADRIANA: The question I want to ask around Flex Fridays...because even when you have that time dedicated, allocated to doing your own thing on Friday, do you ever feel tempted or pulled into the other day-to-day management-y things, and if so, how do you protect your time? MARYLIA: Oh, that happens. Since it's the day that I don't have any meetings and it's less likely that people actually message me, I was like, you know what? Let me just...and I actually take the day because we have, for example, on our projects, we have a column for quick wins that are things that you can finish in an hour, a couple of hours that we don't put on our roadmap, just improvements.  Sometimes I just go there, and depending on the...if I have a really heavy meetings week, I'm like, okay, I just need to code a little. I go to that column, and I just keep going. [laughs] Like, open 10 PRs in a day, like, fix, fix, fix, just get [laughs] the coding in a little. But then I usually get in the mornings to focus on looking, for example, at that tab that I mentioned, so I try to read one or two from that one, and then in the afternoon, I focus on the coding part. ADRIANA: Well, I think it's awesome that as a manager, you get to still stay hands-on because I think it can be really overwhelming sometimes as a manager or tempting even to deal with the other stuff, the non-technical things which can eat up your time. So I think it's awesome. And I think the takeaway from this is you have to make the time. MARYLIA: Yeah. So I really organize because when I said that, okay, I want to be a manager, but I said that I don't want to stop coding because I really enjoy coding. And I know a lot of people want to stop, and that is completely fine. But I was like, okay, I don't even want a really big team because it would mean that I will have more time managing this and not enough time for coding. So that was even part of the agreement, not having a super huge team [laughter], and then I can focus. So I have like six people under me now, and then I just focus on the day.  So, for example, Mondays is a little bit of planning, and I do a little coding. My Tuesday is one on one, so it's kind of like back-to-back. And then Wednesday and Thursday is again dividing through the day, a little of coding, a little related to planning and management. And then, usually on Friday, it's more focused on the coding. But I'm really good at context switching to the point that I am on a one-to-one, and if you finish early and I have five minutes to the next, I'm going to do a little fix. So I can really switch very easily. ADRIANA: That is very, very cool. I wish I could have that skill set. That sounds amazing. [laughs] ANA: The concept, like, the ability to context switch all the time, the privilege too in a job to be able to do that, I personally love it. And then to know that your brain is always able to kind of jump in and get like the adrenaline of something new and then be like, okay, done. But I definitely can't do it in five minutes. MARYLIA: [laughs] On my Chrome, I have groups. So I have one tab like one group for follow, so that is things that my team is working on that I'm looking into, just reviewing, doing stuff. I have one for planning, so all the things related to I need to create time, or these are the things coming in and people are asking me and creating milestones. Then I have the tab for the Friday reads. I have a tab for current, everything that I'm currently working on. So on my day, I kind of go over, depending on the time of the day, to a particular of one of those groups.  So I start with looking at the following just in case people have something that they're waiting on me, do the reviews, then go to my current tab and focus there. Depending on the day, go to the planning. So I keep [laughs] changing between those groups. ADRIANA: That is awesome. That is super badass. I think we've got a mini management 101 going on here. ANA: [laughs] I think you're going to need to follow up this podcast episode with a little blog post on how you stay productive as a manager and IC at the same time.  MARYLIA: [laughs] ANA: It is a lot to manage. It's hard to be context-switching all the time. But I know when I talk to a lot of managers, there's a lot of overwhelm. In general, the job is very draining and overwhelming due to having to know a tech stack and then also being a people person. So it's just kind of hard to balance. And we do have technology. We work on technology. There are ways we can use these solutions to make our day-to-day a little bit easier, and we can kind of do our job better. MARYLIA: Yeah. I'm talking because I've been lucky that I'm really enjoying the things we're doing. But of course, there are times that a lot of things are happening at the same time, and you're just like, ah, I just need a break. [laughs] Again, I have, for example, a lot of empathy. So when things are happening with people in my team, I get really affected by it.  And I have to remind, oh, it's not something with me, it's just something that is going on, or else I'm like, oh, I should have changed this to be better. So I get affected a lot by that. And it's still a little hard for me to put that line that here are things that I can control and after that, I cannot control. That is something that is still very hard. ADRIANA: I really feel you. And I think this is a great segue to our last question, [laughs] which is, what are the things that you do to take care of yourself? MARYLIA: So I have the time for work, and I really try when I'm outside of work to not try to think as much. And even things like taking care of my health. So I go swimming every morning for like one hour. So I start the day with swimming, then I go to work. And then on weekends, like, play video games, watch a bunch of TV. [laughter] I don't want to think. I just want to sit down on my couch and not use my brain [laughs] just to unwind a little. Even the really heavy week, you're just like, I don't want to talk with anyone now. [laughs] ANA: [laughs] I shut down as a human, and I become a vegetable on the couch, and do not talk tech to me. MARYLIA: Yes. [laughs] ADRIANA: It's great that you give yourself permission to do that because I think it's very hard. I need to work on giving myself permission to switch off from work. [laughter] I think it's great. I think it's a really good habit to switch your brain off, become a vegetable. MARYLIA: [laughs] Yeah. One thing that I also did is when I was working at my previous job I had things like Slack on my phone and things like that. So one thing I did is like, at least on Android, you can have different profiles. So I have a personal and a work one.  ADRIANA: Nice. MARYLIA: So all those things related to my email from work or tech, I put it on that one. So I don't get notifications at all. Depending if I know that something might be going on, I was like, okay, I can check, then I go to that one, check a little if there's something going on, and then I go back. So it's not something that I'm constantly looking and checking my messages. ADRIANA: And I think that's so important is being able to switch off those push notifications so that they don't drive you nutty. I feel like after a while; I had to switch off all my notifications for my day-to-day because otherwise, I felt like I was getting PTSD. I was just jumping at every single little thing. I don't get email notifications. Every so often, I'll check my email.  And I think the only thing that I have running during the day are my Slack notifications, and even that is not on my phone, just on my computer. And it's very specific, like, if you @ me or if we're having a DM [laughter] because otherwise, it's so easy to get so distracted by all these little dings. So good on you for putting those boundaries in place. And I think that's really awesome advice for our listeners. ANA: I'm a firm believer of trying to keep that hard line between work and personal stuff. So the idea of putting work apps on personal phone has always been hard for me. And of course, due to the nature of my job of being on social media and being on the road, it always kind of eventually comes back, and you're like, oh, how did we get here? Like, I only had Slack for that one conference, and all of a sudden, [laughs] I have all 20 work apps on my phone, hmm. This was an oopsie. [laughter] ADRIANA: And you have to pare down again. [laughs] ANA: Yeah, I guess I should just put out like a bi-monthly reminder of, like, check in with myself [laughter] of like, hey, what do you have for work on your phone, anything you can remove? [laughs] MARYLIA: Or just create something that, from time to time, you just wipe off everything from your phone, so if you really need it, you have to -- [laughs] ADRIANA: I've heard of people who'll go on vacation and delete certain apps from their phone, like work-related, which I think is a really good habit. [laughs] ANA: I have two things that I always say, if you're going on vacation, you cannot take your work laptop. I understand emergency mindset and stuff, and it's just like, you will not disconnect. You're going to be very tempted. You're still tied to that work luggage. And then secondly, it's that, like, I will delete Slack. I will delete Twitter because, just the tech industry, you're going to find yourself trying to go back to the habits that you have back home of just; like it won't hurt to check like, what is my team up to? What are they saying on Twitter? MARYLIA: [laughs] ANA: And you're not unplugging, and your brain needs to unplug in order for you to bring your best self forward to your team. MARYLIA: When I'm on my vacation, I basically tell my team, "In case there is any emergency, call 911. Why are you calling me? It's an emergency." [laughter] ANA: Yes, way to do it. ADRIANA: I love it. [laughter] ANA: An emergency means you got to call 911. Like, everything else can wait. Is your life in danger, or someone's life is in danger? Pretty sure 911 can take care of it better than I can. [laughter] I am not on-call this week. ADRIANA: Yeah, yeah. I completely agree. That's awesome advice. ANA: I love that we got to end on such good advice of take care of yourself because as an industry, everyone takes care of themselves differently. And the more we talk about it, the more we show folks how everyone else is doing it. Like, we all move forward and being a little less workaholic culture as a generation.  ADRIANA: Yeah, totally agree. ANA: Well, with that, thank you so much for joining us in today's podcast. We had a great time chatting about observability management, moving to new countries, and being Latinas in tech. [laughter] Don't forget to subscribe and give us a shout-out on all social media via On-Call Me Maybe. And be sure to check out the show notes on oncallmemaybe.com for additional resources and to connect with us and our guests on social media. For On-Call Me Maybe, we're your hosts Ana Margarita Medina... ADRIANA: And Adriana Villela, signing off with... MARYLIA: Peace, love, and code. ADRIANA: All right! ANA: Whoo.  [laughter] ADRIANA: We did it.

About the guest: Iris Dyrmishi is a Tech Platform Engineer at FARFETCH. She is passionate about building systems that leverage Observability to ensure their performance, scalability and reliability. Recently, she has been working with OpenTelemetry and exploring how her organization can use it to improve our observability platform.  Find our guest on: Instagram LinkedIn Find us on: On-Call Me Maybe Podcast Twitter On-Call Me Maybe Podcast LinkedIn Page On-Call Me Maybe Podcast Mastodon On-Call Me Maybe Podcast Instagram On-Call Me Maybe TikTok On Call Me Maybe Podcast YouTube Channel Adriana’s Twitter Adriana’s Mastodon Adriana’s LinkedIn Adriana’s Instagram Ana’s Twitter Ana's Mastodon Ana’s LinkedIn Ana's Instagram Show Links: FARFETCH AWS OpenTelemetry OpenTelemetry Helm Charts Jaeger Tracing Prometheus Grafana OpenTelemetry Collector Tail-Based Sampling Distributed Tracing The Evolution of Game Days   Transcript: ADRIANA: Hey, y'all. Welcome to On-Call Me Maybe, the podcast about DevOps, SRE, observability principles, on-call, and everything in between. I am your host, Adriana Villela, and with me, my awesome co-host... ANA: Ana Margarita Medina. ADRIANA: And today, we have Iris Dyrmishi, who is an Infrastructure Engineer at FARFETCH. Welcome, Iris.  IRIS: Thank you.  ADRIANA: So first things first, we're switching things up a little bit today. We'd like to know where you're calling in from. IRIS: I'm calling in from Porto, Portugal. [laughs] ADRIANA: Awesome. IRIS: I think that's pretty far away from where you're at, Adriana. Where are you calling in from? ADRIANA: I'm in Toronto, Canada. And Ana? ANA: I'm here in the San Francisco Bay Area, California. ADRIANA: So we are spanning three time zones today. [laughter] I love it. That's one thing I like about the show is that we get to talk to folks from all across the globe. Classic On-Call Me Maybe question for you, Iris, is what are you drinking? IRIS: Today I'm drinking a caramel boba with rainbow toppings, so I'm very happy with my drink. [laughter] ANA: You said the magic words. You said boba. [laughter] And I also love that it's rainbow toppings, like, [laughter] hell yeah. ADRIANA: Rainbows make the world go round. [laughs] How about you, Ana, what are you drinking? ANA: Today, I decided to pick up one of the Starbucks strawberry refreshers. And I do have a long day of work, so I just ordered a trenta-sized drink which is bigger than my face. [laughter] ADRIANA: It's a big drink. [laughs] I don't think I've ever known anyone ordering one of those. That's cool. [laughs] ANA: I will only do it for the teas. Like, if I know I need to hydrate and want a flavorful, sugary drink, I was like, it's worth the money. ADRIANA: Yeah. Can you imagine that much coffee? [laughs] You can be jittery by the end of the day. [laughs] ANA: I actually think they don't sell coffee drinks in trenta size. I think I looked into it once because it's like a heart attack waiting to happen. ADRIANA: Yeah, I feel like I'm feeling a heart attack without even having had that. [laughter] ANA: What are you drinking, Adriana? ADRIANA: Today, I had the presence of mind to make myself a matcha green tea. So I'm very excited to have something a little bit different other than water. [laughter] ANA: Yums. Always got to stay cozy. ADRIANA: Yeah, totally. It's kind of a coldish day in Toronto. We're supposed to be expecting a big blizzard later today. [laughs] So this is my pre-emptive coziness. ANA: [laughs] Blizzards are never fun. And, I mean, I think that's kind of been the interesting part where we have all these weird climates going on right now. Everyone is kind of in some form of storm around the world; it seems like. ADRIANA: Yeah, it's true. How's the weather in Porto? IRIS: Well, it's pretty sunny but also very chilly because of the Northern stream. It's always very windy here.  ADRIANA: Oh. IRIS: So the temperature might be around 12-13 degrees, but it feels very cold because of the wind. ADRIANA: It's like a windy San Francisco day. [laughter] ANA: Sounds very normal to my life. [laughter] IRIS: There is no beach for us. That's for sure. [laughter] ANA: Well, Iris, the first question that we wanted to ask you for today's podcast is, how did you get started in technology? IRIS: It's actually a very interesting story because, okay, I've been a tech nerd, let's say, since I was a little girl when I got my first computer at 9. I loved taking computers apart, [laughs] breaking them, fixing them. ANA: Hell yeah. IRIS: And when I was about to go to university, I had my mindset that I was going to study computer science. I come from Albania, actually, the other part of Europe. And I applied for university, but then I received a scholarship in another university in Bulgaria, [laughs] so I decided to go there. And I said, okay, computer science is too hard, so I'm not going to be able to maintain a good GPA. I'm going to go for something easier. And economics was the first thing that came to my mind, so I switched from a tech person to economics.  And then I had this very good friend I have actually, and she gave me the pep talk: never forget your dreams [laughs] no matter how hard it is. And it actually worked. [laughs] So I enrolled in computer science classes, and I never looked back. I'm very happy that she convinced me. [laughs] And I did keep my GPA, fortunately, so yeah, there were no issues there. [laughs] ANA: That is awesome. It's always nice when you have people that kind of give you that push of like; you can do this, like, you are capable of it. IRIS: Yeah. And, for me, you know, there are some people that say that "Oh, if I didn't have to earn more money or if it wasn't difficult to get a job in another area, I will probably be doing that." It's not like that. For me, it's technology. My passion is technology, and I'm actually doing it. So it's amazing. [laughs] It's a great combination. ADRIANA: Oh my goodness, living the dream. That's so awesome. So with your computer science degree, how did it lead you to your current career path? IRIS: Actually, in university, I majored in computer science with a background in software engineering. So I worked for three months as a back-end developer. [laughs] And then the company that I was working for needed people to switch to DevOps and to train them, so I was like, okay, [laughs] so I went there. And yeah, that's how it started for me.  We were doing small work in different customer companies setting up monitoring for them. And that's when I started becoming very curious and liking that sort of thing. And then, I switched to the job that I'm currently at. And we're actually creating our own platform. And it just got from the small pleasures of setting up small monitoring here and there to building a full observability platform. So yeah, [laughs] that's how it started, and that's how it's going. [laughs] ADRIANA: Wow, that is super exciting. And you got into...so it sounds like you got into DevOps pretty early into your career, too, which is cool. IRIS: Yes. In my university, surprisingly, usually, in computer science degrees, DevOps is not really taught. So when I entered, I was like, what is this? [laughs] Like, what are these things that I never heard about? [laughs] So it was scary. But it's a good thing that I started when I was uncomfortable in the field altogether. I wasn't confident enough in software engineering either. [laughs] So I got into that pretty easily, and it was just a matter of learning. ADRIANA: Very cool. It's interesting you mentioning that your university didn't teach anything about DevOps. And I think we've had similar conversations with other guests about there are certain things that you do out in the real world once you get your degree in computer science, computer engineering, whatever [laughs], but you're not taught this stuff in school. I think we've had a guest recently, Michael, who one of his first jobs out of school was SRE, which is kind of mind-blowing when you realize the fact that the only way to learn SRE is by doing it. And similarly, [laughs] for you, the only way to learn DevOps is by doing it.  IRIS: Exactly. [laughs] ANA: It's definitely an interesting trend because I do think we're starting to see universities pick up on people are not coming to our colleges because the education they get in computer science does not place them in a job right out of school because we know the interview loops for these jobs tend to be pretty tedious. And I think we're starting to see a lot more schools start to realize, like, we actually need to fill in that gap. ADRIANA: Yeah, yeah. I really do hope to see more of that gap being filled in real life because we're definitely, yeah, we're still hurting. Circling back to a point that you made earlier when you were talking about building up a platform at your current workplace and getting to dig deeper into observability, can you tell us a little bit more about your observability journey? IRIS: Let me start from the beginning. So my observability journey, as I said, started with just setting up. AWS was my weapon of choice. [laughs] That's where I practiced my first observability and basically just collecting metrics and in cloud setting up dashboards, setting up alerts. That's how it started. Then it progressed to building a full platform, basically orchestrating applications, building the infrastructure to actually scrape the metrics, collect the traces, and basically the logs as well.  I don't know if I'm explaining myself properly in explaining the platform. But it went from actually using what we had, what was already generated by default, to building a platform that allows the teams to set up their own metrics and basically building tools to allow every team to be able to set up observability for their own application because they know it best. And what we do is we provide all the support that they need for infrastructure wise and all the guidance when it comes to metrics or traces, the guidelines about configurations, and stuff like that. ADRIANA: Nice. I love to hear that because I've always been railing on people who are asking the teams who are not involved in the development to instrument their code. And I get so mad because it's like, [laughter] it makes no sense. I don't know your code. How can I instrument this? [laughter] It's very nice to hear that your team is providing guidelines but not doing the work for them because it's not really how it's supposed to work. IRIS: Yeah. I believe observability is a responsibility of everyone. It's not just, for example, five people setting up everything for everyone. Every engineer has to do their part. We, of course, have to provide the tools to make that happen and to make that very easy so engineers are not spending all their hours in a day to set up monitoring and observability but that should be their responsibility as well.  ADRIANA: Yeah, yeah, absolutely. ANA: And it also just makes you a stronger engineer. I'm learning that skill set of being able to know how to instrument your code, how to be aware of the resources that it's actually taking up, to knowing that if you want to move up the career ladder and switch jobs or anything like that, that this is actually something that you are aware of. And you're not in that space where you're just throwing it up against the wall and hoping that it works and your customers are happy. IRIS: And, at the same time, troubleshooting is a lot faster if you are the one who has set up the monitoring and observability for your application because if someone else did it for you, you will, of course, have to go to them for help to help you figure out your own application and your own code, what it is causing and why [laughs] it's breaking. So that's not ideal. [laughs] ANA: It's like you and your team are going to know what functions you deployed in the last release versus the SRE team that has to manage ten services for the entire platform to work. Like, [laughs] why is there an increase in HTTP 400 errors for this one service that just got recently rolled out? [laughs] So, since you've been working in the space of observability for a bit, what do you say is one of the things you enjoy the most or your favorite? IRIS: One of the things that I love the most is how fast the area of observability is moving. I feel like it's one of the areas with the most modern technologies. It's always building, always improving. For example, I work a lot with Prometheus. There are always new features coming, and currently, I'm working with OpenTelemetry, which I am in love with.  ANA: Yay. [laughs] IRIS: And you can imagine it's the most modern technology. You get your hands on some of the best things out there. So that's one of the things I love about observability right now. I don't know what's going to happen in five years. [laughs] I hope it continues the same. ADRIANA: That's awesome. OpenTelemetry is near and dear to my heart. I got involved, I guess, more heavily involved as a contributor to OpenTelemetry after I joined Lightstep last year. Same as you, it's been super cool to watch it evolve. I would love to hear about some of your experiences with OpenTelemetry, the good and the bad. IRIS: We are pretty new to OpenTelemetry. First of all, I heard about it last year. And I just played around with it a little bit, and it caught my attention. So now we actually had the space to work with it and the need to work with it. And I have to say I love it for the fact that, first of all, there is such big community support in OpenTelemetry, so it is very easy to actually implement it into an already running solution. You always find the Helm Charts are extremely helpful. There is very good documentation.  And the other thing that I love about it is that it allows us to solidify our platform in one. Instead of using several tools, for example, logs, traces, metrics, we just use OpenTelemetry.  ADRIANA: Nice. IRIS: But my favorite part so far is how easy it is to just fit it right into an existing architecture and make it work without having to change everything [laughs] and spend months and months of work on it. ADRIANA: That is really, really cool. I'm super stoked to hear that. What are the ways in which your own team uses OpenTelemetry? Do you guys stand up collectors? Or do you provide guidance on instrumentation to other teams? Like, what kinds of things do you guys do?  IRIS: So, my team, we're responsible for the collector and the infrastructure in general. We have another team that is helping with the instrumentation. We work closely together, but my team works with our infrastructure. We are very close to setting it up in production, and I'm very excited about it, [laughs] the collectors. And then, we're going to start with the agents as well to make a full OpenTelemetry solution. [laughs] Currently, we're using Jaeger. So we are going to try and put them together until we have a full OpenTelemetry solution. ANA: Nice. It's always nice when you kind of get to bring things together and start finding the entire space of observability. Like you said, being able to now have your traces, logs, and metrics all in one with OpenTelemetry is kind of like a nice little package. IRIS: Yeah. And I have to say backwards compatibility as well is amazing. For example, I mentioned Jaeger earlier. It's just amazing [laughs] that we don't have to switch back and forth to make things work, and that's great for me. ADRIANA: Yeah, yeah, totally. I think for me; the aha thing was that you're not locked into a particular vendor. So if you decide, hey, this vendor is working great for me now, but two years from now, nah, it's not working so well, you're not cussing [laughs] over the fact that crap, I got to re-do all this crap. And your developers are also not super mad at you after they've spent all this time instrumenting their code, and you have to be like, no, sorry, you have to use something else. [laughter] That's not the case with OpenTelemetry, which I absolutely love. IRIS: Yeah, absolutely. [laughs] ANA: I feel like we need a lot more tools in this space that have implementations like that, like the vendor agnostic open-source projects, because it is true, like, the amount of heavy lifting that a team or an individual needs to do because it's like, oh, sorry, this contract is going up by like $500,000 a year. A company can't support it anymore. And it's like, well, in order to rip this apart and go build it with another vendor, that's going to be another 500,000 anyway. So what is the company going to do? [laughs] IRIS: Yeah. I can't imagine that for a big company that has engineering teams working with different technology, different programming languages. It must be a headache to switch. [laughs] ADRIANA: Yeah, seriously. I feel like that would be a nightmare. So you guys, when you started implementing OpenTelemetry, was there anything already in place as far as implementing distributed tracing or metrics, like, any of the signals, any of the observability signals? IRIS: Yeah, we do. We use Prometheus. We use Jaeger. We had basically a full monitoring platform with Prometheus Internal Grafana alert manager for alerting and, of course, Jaeger for tracing. So we did have existing platforms. And that's why OpenTelemetry seemed so attractive because we could get it right there without disrupting the whole thing. We work with many teams, and there are a lot of traces and a lot of metrics being collected.  So we wouldn't take the risk of switching to another tool unless it really fit and it actually benefited us in this case. Because, yeah, Prometheus is an amazing tool, and we wouldn't just go and disrupt it. [laughs] A lot of people would be angry [laughs] if that happened, not being able to have observability at all. ADRIANA: Yeah, I totally agree. So because of your existing stock, I guess it sounds like you guys didn't have to rip anything apart to be able to leverage OpenTelemetry right away. IRIS: Yeah, yeah, exactly. We don't because it basically supports all the tools that we're currently using, and we build from there. We don't have to have a full solution and remove one and put the other. We can put it piece by piece, and we will have full availability 100%, and at the same time, implement OpenTelemetry until it's a full OpenTelemetry platform. ADRIANA: Nice, nice. ANA: No work goes wasted, which is always nice. [laughs]  IRIS: Yeah. ADRIANA: Now, keeping in with the observability theme, I think in our pre-chat, you mentioned having some experience around tracing. And, of course, you mentioned that you all use Jaeger at FARFETCH. Tracing, I feel like, is one of those controversial topics for folks getting into observability because they don't necessarily see the value of it. What are your thoughts around that? And share some of your experiences on tracing. IRIS: I have to say it infuriates me. I'm a big [laughs] tracing fan, like a diehard tracing fan because...that infuriates me because I know there are so many engineers that say, "Oh, why do we need tracing when we have logs?"  ADRIANA: Yes, yes.  IRIS: And it's very important to make people understand what is the difference between them. And I feel like the job of observability or an SRE team is to also educate on these topics and help people understand and engineers understand how important tracing is and how different it is from logs. So one thing that is with tracing is that it can be pretty expensive to have traces fully for teams. And that's why sometimes it is seen as something that, oh, we don't need it because not everything is saved; for example, we could have a 2% sampling size, 10%. It's never enough. Some spans, of course, might be dropped by the collectors. And people are like, oh, but this is not enough information. So I can understand the first one but logs and traces no. [laughs] I am completely against it.  But this one, I understand because we can never get full sampling or maybe, yeah, if we spend a crazy amount of money. So people see that as a negative side to tracing. But there are ways that can be improved, and that's why I come back to OpenTelemetry. As I said, I'm very passionate about it [laughs], and tail-based sampling that could help. There are ways to make this happen. But anyway, every information that is in the traces can be very useful; even to see the flow of a call or the flow of an application can be very useful for an engineer. Even if it's not what they were expecting to see, at least it will take them to where the issue was. ANA: What do you say to that engineer that doesn't want to spend the time to instrument their code in order to have traces that's just like, nope, I'm not doing this work? IRIS: What I would say, honestly, I have never had a conversation with an engineer that is against tracing. I feel that right now, engineers are very well educated about observability because it's becoming so big, but I'm sure there are some. The only thing that will be is your loss. [laughs] I'm joking, of course. But what I would say is that it is really your loss because you're going to miss out on a lot of information and in the troubleshooting moments because when the application is running perfectly, maybe you will not care about the traces.  But when you need to troubleshoot something, that's when you will see the real value of tracing, so try it and see. [laughs] Or I would also offer to have a game day. Okay, let's make your application fail and see how much tracing is going to help you. Show with data and not just an opinion because it can be considered my opinion. [laughter] ANA: I love game days. I love that you brought them up. [laughter]  ADRIANA: I was going to say this is like your love language. [laughter] ANA: Literally a love language, like, inject failure into your infrastructure, into your life. Think about what could happen and move forward with those learnings that you have. I think a lot of people don't realize where that concept of failure and observability go hand in hand. I'm actually writing a talk around this right now about you make the end goal reliability, but you don't ever care about the steps that you need to do in order to get there. So I'm trying to tell folks make observability; your goal and reliability will follow. Like, it all kind of goes hand in hand. IRIS: I'm happy it's becoming so big, [chuckles] the observability topic because it's very important, and it wasn't understood as much before. It was seen as a side thing, oh yeah, maybe observability or not.  ANA: Do you have any tips for folks that are embarking on their OpenTelemetry observability journey where they're coming from a place where they haven't necessarily done that?  IRIS: Keep breaking things [laughter], and that's the only time that you're going to actually see success. [laughs] Because I know many people that when they try a new technology, they always try that locally, and it works amazingly. But from trying something locally and deciding whether it works for you or not, whether it's easy to implement or not, and going to a thriving dev environment or a staging environment is completely different.  So my advice is try and put that infrastructure in a live environment, not production, of course, but in a dev environment, when you have real applications running, you have connectivity; you have all those things. So you can actually see how much it's going to take you to implement it and how much value you're getting for it; only then you can be fully confident with it. ANA: Is there any way that you have found to quantify that value or to explain it to your leadership? Because I think that's sometimes something really hard in order to get, leadership to buy in so that you actually, one, have the time to do observability, to work on projects like OpenTelemetry. How do you start telling leadership like, hey, this is actually going to help us in the long run? IRIS: This is surprisingly not an issue. For us, OpenTelemetry was always in the table and actually brought to us by leadership.  ADRIANA: Wow. IRIS: Of course, our intentions were there as well. So I feel like, as a company, we are very well educated to the new technologies and the benefits it can bring us. So I'm very happy about that, very, very happy actually. [laughs] ANA: That's awesome. It's sometimes really hard to find that. When I talk to most folks, they are just like, whether it's observability or reliability, they're just kind of like, well, as engineers, we really want to bring in this technology. They're going to conferences, hearing the talks, learning about all these tools. But leadership is just like, no, we got no time. We got no money, like, [laughs] no. IRIS: I don't know. Honestly, of course, the leadership goes in layers because it's a big company. But I don't want to brag, but other engineers that I work with are some of the most knowledgeable and know a lot about the tech scene right now. So it's very easy to get these new technologies approved and to work with new technologies brought to you to see and to try and to test it. It's very nice, actually. ANA: That's exciting. ADRIANA: That's so refreshing. It's nice to hear that there are companies out there that are thought forward. [laughs] ANA: As you've been working on making observability better, has it made you and your team start thinking about other things you want to improve on alongside the platform or just the way that you engage with other teams? IRIS: Yeah, we're always trying to find places to improve. First of all, we want to improve on performance as well because, of course, the more time passes, the more teams join and the more information we have to collect. So our main goal currently is to work and to improve performance, therefore, improving the cost as well. We cannot just let the platform run wild. [laughs] So yeah, we are trying to find a good solution there.  And always, well, I would call them customers, but our engineering teams we try to work very closely with them. But yeah, we're always trying to find ways to better communicate with them. As I mentioned, sometimes it could be that the observability team is not seen for what it really is, that it's providing you the tools to actually do the work and implement observability yourself. So that's something where we have to work more on and try to communicate more clearly to the teams and show them. But it's not something that happens very often, but definitely, something to improve because the more teams are added, the more we will have to have this conversation, basically. ANA: Definitely.  ADRIANA: But as far as at least explaining the value of observability, it sounds like that's all sorted out. It's just now how to execute on it is more of, I guess, the challenge. IRIS: Yeah, exactly. I think it's important for an observability or SRE team to be there for the teams to help them figure out how everything works and how to set it up. As I said before, of course, our job is not to go and create dashboards or set up metrics or traces for an engineering team or for an application, but it is to guide them. And at the same time, I work with observability every day. I work with those tools every day, so I know them by heart. Even in my dreams [laughter], if you ask me a question, I will answer about observability.  But we have to understand that there are other engineering teams that work with completely different technologies, so they're not aware of how everything works. So we have to be available to help and to lend a hand as long as it is a collaboration and not us taking all the load of setting up observability for them. ADRIANA: I think that is so perfectly put because I've been in situations also where I was on an observability team, and we kept getting asked to create dashboards for other teams. [laughter] And it's like, we can guide you, but this is just weird. [laughs] I don't know your stuff.  IRIS: Exactly. ADRIANA: So I'm glad, like, emphasizing the two key things. Y'all instrument your own code. Y'all create your own dashboards, but that you're there to support and guide them and to collaborate with them. I think those are key things and I think excellent takeaways for our listeners, especially those who are embarking on their observability journey, because I think that's where a lot of organizations can end up going wrong. So these are really good practices to follow. IRIS: Absolutely. ANA: As you've engaged with these customers, the engineering teams, do you have office hours set up, feedback forms, Slack channels, what is your actual way of serving them and letting them come to you? IRIS: We have, of course, Slack, which is always open communication, but people know people. So, for example, there are many people that know who is in the observability team; in this case, some of them know it's me, so they come to me, to my co-workers, and so on. But, of course, we have official ways like service request, feature request that teams could ask for assistance. And we will help them no matter what it is, even if it's, for example, an incident or if it's a new feature that they would like or some new technology that they would like us to look at because it might be beneficial for them. We're open in many ways to accommodate these requests. ANA: Have you or your team ever been called into an incident just as support?  IRIS: Yes, it happens a lot, but mostly when something from our site is broken, for example, in this case, metrics. We could use metrics for autoscaling for other functions. So it is important that the metrics are always running well and configured properly, all that. So if, for example, that happens, we are called. But I don't think there was a case that we were called when another team's application went kaput [laughs], and we were there to provide assistance because we are the observability team. I don't think that happens. And that's why I was saying earlier as well. I feel like we have a very good culture currently in the company when it comes to observability. ANA: That's really awesome. You got to continue cultivating it. IRIS: [laughs] Yeah. ADRIANA: Yeah, yeah, definitely. You guys are like the poster children [laughter] of how to do observability the right way. I love it. I love it so much.  IRIS: I don't want to brag, but -- [laughter] ADRIANA: No, you should. You should. [laughter] ANA: Brag about it. Tell people that y'all are hiring. IRIS: It's pretty great. I think we're lucky in that regard. We've done a very good job. But it's also, of course, the environment itself and the juniors we work with that make it possible. ANA: You've gotten a chance to share about your observability, OpenTelemetry journey. What do you say is the biggest evolution you've seen since you first started working in this space to where you currently are now? IRIS: I think the biggest evolution has been migrating the whole platform to a more scalable and reliable environment. I think one year back when I joined the team...it's actually not that long, but I'm very attached [laughs] to the team and the mission we have. The platform was not as available and scalable as it is right now. So for us to be able to be so quick on our feet and onboard new teams and new metrics, new traces, it was a lot more difficult.  Currently, we have an amazing platform that is fully scalable and available 100%. And it's much easier for us to maintain but also to improve. So the improvements that we did one year back were much slower than they are now, thanks to the migration that we did. So I'm very proud of that.  ADRIANA: Oh, that's amazing. Just switching back to, you know, we were talking earlier about OpenTelemetry and specifically how you really enjoyed the responsiveness of the OpenTelemetry community. Have you and/or your team had a chance to also make contributions to OpenTelemetry? IRIS: Not yet. I'm very scared [laughs] for some reason to start there, but I really much want it. And probably it's going to happen to have a receiver, sorry, an exporter for Cassandra for the database because currently, as far as I know, it's not supported from what I have seen and from what I've tested. So I'm planning to make my own contribution, [chuckles] but I've been scared so far.  ADRIANA: Oooh. IRIS: I don't know; I find other people that contribute; they are such good engineers [laughter], and sometimes I don't feel up to the level. But I'll get there, I promise. [laughs] ADRIANA: That's totally fair. We had a discussion with one of our previous guests about contributing to open source and how it's so scary because you're putting yourself out there. You're so vulnerable exposing yourself to the world, to the judgment of others. IRIS: Exactly. [laughs] ADRIANA: I can completely relate. I remember my first OpenTelemetry contribution. I'm like; these guys are so smart. Are they going to think I'm an idiot? And for me personally, what helped, and I mean, I know everyone's got their own path, but for me contributing to the OTel docs was a nice way to ease into it. Because I feel like [laughter] nobody is scrutinizing my code. It's just scrutinizing my writing, which I feel [laughter] is a little bit less scary. IRIS: Yeah, that's not as scary. [laughter] ADRIANA: So, anyway, that could be if you want to just dip your toes into OpenTelemetry, that could be a cool way to get into it. But I can say that for all the pull requests that I've made in OpenTelemetry, no one has ever been evil. The comments have never been mean. They've always been very polite. So that has put me at ease a fair bit. So I hope that you also have a similar experience when the time comes. IRIS: Thank you for the recommendation. This is actually very good to ease myself into the documentation and then actually [chuckles] start writing code. Yeah, I do appreciate constructive criticism, but yeah, it's scary still. I pride myself on being a good engineer, and if someone says something mean about it... [laughs] ANA: I think it's that people forget that you can contribute to these open-source projects without it being a full code repo or all these bug fixes or anything like that. There are ways that you can still create that PR and be like, oh look, I'm starting to contribute to such a project that I'm part of.  And additionally, to piggyback off what Adriana was saying, it has been a welcoming community for OpenTelemetry, for other CNCF projects that I also contribute to. I think the umbrella of all those projects, the communities that are being built in this cloud native space, have gotten it right of, like, we're all just trying to build cool shit. [laughter] Like, let's just be nice to one another and remember that we're human, like, we got feelings. We got bad days. We are going to have to fail. We're going to have to learn. It's just that constant reminder. ADRIANA: Cool. Well, I guess we are coming up on time. So as we wrap up, we would love to know your thoughts on what you would like to see in observability in the future. What's part of your wish list? ANA: Well, part of my wish list about observability it will be, first of all, to keep continuing to grow so much and to be so modern and fast-moving because it's fascinating right now. And another one would be for more companies to actually start implementing and appreciating and seeing all the benefits of observability. I think some teams and some companies are there or at least are getting there. But I know that there are so many that still are very far from the goal of embracing observability, of how it should be, [laughs] not the observability that is different from the one we've been talking so far. [laughs] ADRIANA: Yeah, yeah. IRIS: But I really hope that all companies get there because it's going to be very, very good for them and, of course, very good for the observability community and observability engineers who invest so much time and their skills on these tools.   ADRIANA: Absolutely. ANA: Most definitely. I think a lot of learning is going to come out with more movements like that. Well, with that, thank you so much, Iris, for joining us and talking a lot about infrastructure and observability. Don't forget to subscribe and give us a shout-out on social media via On-Call Me Maybe. Be sure to check out the show notes on oncallmemaybe.com for additional resources and to connect with us and our guests on social media. For On-Call Me Maybe, we're your hosts, Ana Margarita Medina... ADRIANA: And Adrianna Villela signing off with... IRIS: Peace, love, code.  ANA: Whoo! ADRIANA: Yeah. [laughter] ANA: Yay.

About the guest: Sasha Rosenbaum is Principal at a new venture, Ergonautic. With a degree in Computer Science, an MBA, and two decades of experience across development, operations, product management, and technical sales, Sasha brings a unique perspective to optimizing the organizational flow of work, bridging gaps with empathy and insight. Find our guest on: Twitter LinkedIn Mastodon Find us on: On-Call Me Maybe Podcast Twitter On-Call Me Maybe Podcast LinkedIn Page On-Call Me Maybe Podcast Mastodon On-Call Me Maybe Podcast Instagram On-Call Me Maybe TikTok On Call Me Maybe Podcast YouTube Channel Adriana’s Twitter Adriana’s Mastodon Adriana’s LinkedIn Adriana’s Instagram Ana’s Twitter Ana's Mastodon Ana’s LinkedIn Ana's Instagram Show Links: Ergonautic DevOps DevOps Days Chicago FileZilla [Video] “Single Person of Failure” by Sasha Rosenbaum Google SRE Book Andrew Clay Shafer Transcript: ADRIANA: Hey, y'all. Welcome to On-Call Me Maybe, the podcast about DevOps, SRE, observability principles, on-call, and everything in between. I am your host, Adriana Villela, with my awesome co-host... ANA: Ana Margarita Medina. ADRIANA: And today we are talking to Sasha Rosenbaum, who is a Founder at Ergonautic. Welcome, Sasha. SASHA: Thank you. I'm so happy to be here. And it's a pleasure to be chatting with you all on this podcast. ADRIANA: Yeah, we're super excited to have you. So first things first, what are you drinking today? SASHA: I'm drinking peppermint tea. And this is a funny story; I recently became allergic to black tea. ADRIANA: [gasps] Oh no. SASHA: Which I previously drank all my life. I didn't even know you could be allergic to black tea. So now it's either coffee or I try to find some type of herbal thing. [laughs] ANA: [laughs] ADRIANA: Oh my goodness. At least it sounds like you've found a nice alternative.  SASHA: Yeah, I have so many variations of herbal teas. It's not even funny.  ADRIANA: [laughs] SASHA: Because I keep trying them all, and I can never quite find one that I'm committed to, so it's just a whole big cabinet of tea. ANA: [laughs] I am sorry for you having to not have access to black tea anymore. I think that's always been my favorite, like, oolong tea/black tea, and especially when you get to combo it with a peach or a mint on top, like summertime. But I'm glad that you can still drink coffee because I know for some folks, it's like they have to cut off tea and coffee. SASHA: Yeah, it's funny because the hardest part is people don't usually have hot drinks that aren't tea and coffee. So I'm like, if it's like 8:00 p.m., I'm not going to drink coffee, so I'm stuck with water. ANA: [laughs] Fair enough. ADRIANA: Yeah, that's sad. That's sad. ANA: I'm very close to Sasha's drink. I'm having something called a mint mojito. There's a coffee shop in the Bay Area called Philz Coffee. And this is one of their most common drinks where it's like you get a latte, but they also muddle mint into it, and they make it really sweet with a lot of foam. So that was kind of what I was feeling this morning. ADRIANA: Nice. Nice. I did not go with anything creative. I just have my water here. ANA: And it's too early for boba tea for you. ADRIANA: I know, right? Yeah, I think the places don't open until at least 11:00 if I'm lucky. Otherwise, I have to wait till noon or make my own. [laughs] SASHA: Oh no. Oh, can you make your own? I've never tried making my own boba tea. ADRIANA: It's not bad. Like, you can get the pearls on Amazon, and you just boil them for like one or two minutes. And if you want, you can soak them in honey or brown sugar, and it's actually pretty good. SASHA: Nice. You know what I like? The new ones are like jelly, so lychee jelly or something like that. And it's just like, I like it more than boba because boba is so giant. [laughs] ADRIANA: Yeah, that's true, and it's filling too. They pile so much on. I'm like, oh my God, I'm like eating my drink. [laughs] SASHA: It's a whole snack. ADRIANA: Yeah, exactly. ANA: I'm insane, and I like it when it's boba and then lychee like jelly. It's just like the double whammy of it. ADRIANA: It's a good combo. SASHA: Any drink that has lychee in it, I'm all for it. [laughs] ADRIANA: Me too. Oh, I'm such a huge fan. Whenever I find the fresh lychees at the supermarket, I'm like, oooh, give me, give me, give me because they're so, like, it's seasonal here. I don't know how accessible it is where y'all live, but here, I think I can find it in February and maybe one week in the summer. ANA: We definitely don't have too much access to it here. I think I had more access to it in Miami and in Central America. But I want to say in Downtown San Francisco, most shops always carry lychee, which is actually pretty neat because I'm like, I prefer this over anything else. ADRIANA: Oh yes. ANA: Or when you go to froyo shops, they have boba, but it's the popping kind, and it's got little like bursts. ADRIANA: Yeah, that's right. ANA: Then, at Target, I just saw that they're selling pre-made boba, and it comes with those like popping boba. So I bought it for my partner's kids, and we've yet to have it. ADRIANA: Oooh. ANA: But it comes in a little four-pack pre-made boba, and I was like, that's fun. ADRIANA: That's awesome, little family activity. ANA: Yeah. [laughs] For taco night, my plan was we'll do tacos, and then we'll have these as desserts. [laughs] ADRIANA: That's awesome. [laughs] ANA: Well, Sasha, we'll actually get started on some of the questions that we have for you today as much as we love talking to you about your love for herbal teas. [laughter] The first question that I'm sure some of our listeners are wondering, like, how is it that you got your start in tech? We know that you've gotten a chance to work at various companies. But it's always nice to hear how folks were like, ooh, this seems interesting. Let me take a look. SASHA: So I'm actually, like, I'm probably at this point an unusual woman in tech in the sense that I've been here almost 20 years. And I actually have a computer science degree, so it's like a more conventional path rather than switching careers. But I did actually switch. So I was getting the same message in high school as everybody else that computer science isn't for girls and that kind of stuff. And so what I initially enrolled in university was biology, and I wanted to do genetics. And the first year in biology, basically, I realized the entire degree is like growing yeast and such things in a lab. [laughter] So you do genetic experiments, which sounds really, really cool. But to modify these genes, you grow fruit flies, and yeast, and then mice, and stuff like that. And I'm like, you have to proceed to like Ph.D. to like...and the whole time, this is what you do. You show up every day to your lab, and you're like, feed mice something. [laughs] And I'm like, this doesn't sound like a career I want to have.   ANA: [laughs] SASHA: So I kind of went...and my university was extremely technical, so I went through the entire list. And it was like, chemical engineering, physical engineering, industrial engineering, this, this, and this. And then there was computer science. And I did one Pascal class in school that was really basic Pascal, and I was like, I kind of like that. So this was like, click a button, switch to computer science. But then the moment I was there, it was so fun because it was like basically playing games. What I love about computer science is you start with nothing, like, this blank slate, you know, empty editor. And then you create something that actually produces results, and that is just magical. It's super creative. You have a lot of freedom in how you structure things, and what language you use, what architecture you use. So it was always like, super, super fun to do. That's how I get started.  And then, I was a developer for many years. And then I gradually got into...I did some ops in the beginning, and then I basically got into the DevOps movement. So that kind of shifted my career towards more opsy, SRE kind of conversation. I think I've held every role that you can imagine. So I was a product manager for a minute. And I've been in sales. I've been in DevRel, which to some people, it's marketing and to some people, it's not. So basically, I feel like I've been in every department of the organization but always something tech-related. ADRIANA: You talked about getting involved in the DevOps movement. I wanted to dig a little bit deeper into that. And specifically, how have you seen the DevOps movement evolve since you jumped on it? And follow up to that is are you happy with how it's progressed? SASHA: I got involved...I think it was 2013. So technically, the first conference was 2009, so it was probably about four or five years later, but it was still not a thing. So when people talked about DevOps, like, if you look...there's in Google search results how does search for a certain word change over time. And so the search for DevOps was, like, from 2009 to 2014, it was near flat. And then it kind of starts spiking because people got more and more interested in the term. The devopsdays.org kind of spread the word and started popping conferences all over the world.  So I was a speaker at DevOpsDays Chicago, I think, in 2013, and then I became an organizer in 2014. And in 2013, it was the fifth city that did DevOpsDays, and now there's like, I don't know, over 80, something crazy. So it's all over the world. It's everywhere. So I think there have been a lot of changes in how it started, and how it progressed, and where it is now.  What we started with was basically a lot of talk about collaboration and a lot of talk about automation. So it was two things, like, one is aligning incentives, so you stop kind of chasing two different agendas. Basically, you know, and I'm sure your listeners know, that developers want to deliver faster, ops want to keep the lights on, so minimize change. And then there's this inherent conflict between the team. So the first conversation in DevOpsDays, how do we minimize this conflict? How do we make everybody speak the same language, understand that they're incentivized to do the same thing, which is delivering software to their customers successfully? Then it evolved to people started creating roles for DevOps, which at the beginning was a big no-no for us. I had one of my co-organizers refuse to take a job at a company as the director of DevOps until they renamed the role because he was like, DevOps is not a team. ANA: [laughs] ADRIANA: Yes. SASHA: And I'm not going to work here. ADRIANA: [laughs] SASHA: We have definitely lost that fight. DevOps is now a team. ADRIANA: Yeah, that makes me cry. [laughter] SASHA: So I will say on the bright side, the DevOps teams we did kind of create a job. So, in addition to...because DevOps teams don't exactly do what ops team did and they don't exactly do what dev teams did, so it's kind of like this new set of skills that were defined by this movement. The DevOps term also got co-opted by everyone, and their, like, every company selling a product in this space wanted to say they sell DevOps. ANA: [laughs] SASHA: And so every day I read some article saying that DevOps is dead [laughter] and usually to start an argument. Usually, someone defines DevOps as whatever they want, so they're like, DevOps is all about culture, or DevOps is only CI/CD tools, or DevOps is this. And then this is why it doesn't work. It's like, okay if you define the word to be whatever you want it to be, you can argue against the concept and practice. And then I think the first couple years that SRE became popular, I was kind of like, oh, you're just renaming this thing to another term. But I've come around because SRE actually defines metrics that allow you to reason.  So we talked about incentives, and the incentives are great while you can talk, while you can collaborate, and everybody can be in the same room, and they can communicate with each other. But over time, in the larger organization, that's kind of hard to maintain. If you can attach metrics to it that allow you to reason about this conversation, it makes it easier. So SRE did bring a lot to the table as, in practice, you can implement with SLIs, SLOs, and error budgets and all these things that kind of allow you to discuss what the trade-offs are, which is really important.  I think the unfortunate part of it is a lot of organizations just rename the team to whatever the buzzword is. So ops has been renamed to DevOps; it's been renamed to SRE. Now it's been renamed to platform engineering, whatever comes next. And they don't change any of the practices. So how do you expect to get better results? Like, you say you're SRE...one example is so many organizations are like, we are an SRE team. I'm like, okay, what's an error budget? ANA: [laughs] SASHA: And either they don't have an error budget at all. They don't even know what it is. Or they have a pretty slide saying error budgets are this. They don't have it defined. Or the next one is they have it defined, but no one looks at it. And if you blow the error budget, no one stops development, and they just proceed as normal. I'm like, so you're calling yourself SRE based on what exactly?  ADRIANA: Right. [laughter] SASHA: It's like, I'm monitoring alerts [laughter] like we did 20 years ago, oh no. ADRIANA: It's like fancyops at that rate, right?  SASHA: Yeah, yeah. There's a lot of fancyops around it. And so I had this moment, I don't know, three years ago, whatever, somewhere during or before the pandemic.  ANA: [laughs] SASHA: I had this moment where I came to my co-organizers of DevOpsDays Chicago, and I'm like, "I feel like we didn't do anything. We're having the same conversations for ten years. Nothing changes. Everything is sucking in the same way." And they're like, "No, no, no, you're wrong." My job used to suck. Like, my job used to be I had a once in six months deploy. It took me all weekend. I was on-call for it at 2:00 a.m., and no one cared about my well-being. And developers were like, "We don't care about operations and whatever."  And now I get to implement tools and practices, automate a bunch of this, get releases every day, which makes it less painful, all of these things. So we improve the job of operations a lot. That being said, there are so many, like, if you look (I had it somewhere in my slides.), if you look at statistics from StackOverflow, they did a survey on deployment practices. And the last time they did it was 2017, and they stopped doing it, unfortunately. But I think in 2017; it was something like 20% of all companies deploy through uploading zip files. [laughter] I'm like, where are we? You don't have a single pipeline? You don't...what? Okay. [laughter]  ANA: Like we bring back FTPs and just dragging and dropping like no sense of version- control apart from just dropping your code in some FileZilla. [laughs] SASHA: I've literally walked into organizations where they actually didn't have source control. So they're writing software, and they actually don't have source control, and this was like two years ago. To some extent, it makes me mad when people are like, "Oh, DevOps is such a solved problem." or like, "Everybody's doing CI/CD and deploying everything." [chuckles] I'm like, no, this is not what's out there. There's such a different landscape. I love this quote, like, "The future is here, but it's not evenly distributed."  ANA: Yes.  ADRIANA: That's a good quote.  SASHA: There's such a different landscape with people doing the best best practices of everything and then the worst practices of everything. And then I feel like the problem with emerging tech is people continue to try to sell the new, shiny thing. So an example...and I don't love picking on particular things but companies that sell chaos engineering products...this company can't even deploy code, and you're trying to sell them chaos engineering. [chuckles] Let's go back to the basics. Like, what are we chaos engineering here? [laughs] ADRIANA: I do feel, though, in large companies, I mean, large and small, really, like there are VPs of tech or CTOs who fall prey to the shiny new object as well and then end up leading the company [laughs] in this path of like, let's implement this tool, and, as you said, but their house isn't fully in order. So how can you implement this tool when the foundation still sucks? SASHA: Part of this is how people get budgets to do things. So part of our job at Ergonautic we do management consulting in IT. So we kind of come in and try to help organizations improve their IT practices. And one way to phrase it is like, we'll come in and rescue your DevOps transformation.  ANA: [laughs] ADRIANA: Love it. SASHA: Because you failed, or you think you failed. And we'll come in, and we'll, like, you definitely improved some things, but you're probably hitting some bottlenecks. And we'll come in and just take a view of the landscape and figure out where you're stuck. So coming back to how people get budgets, people got budgets for Agile transformation, then they got budgets for DevOps transformation and for cloud transformation. And you can't sell DevOps twice, so you're bored.  ADRIANA: [laughs] SASHA: So you need a new transformation, so now there's microservices. After microservices, now there's going to be platform engineer, I don't know. So basically, there is this inherent drive to pick up the new, shiny thing. And a lot of times, it's well-intentioned. People are just trying to get rid of technical debt, and that's the only way they can pitch it. It's like, oh, we're going to rewrite everything, so we're on microservices. It's like, no. ANA: [laughs] SASHA: You actually just need to rewrite everything that's 20 years old, and you can't maintain it anymore. You don't need microservices. Your monolith is fine. It's an application that serves 100 people. You don't actually need Kubernetes for it. [laughs] It's fine. ANA: And you also have those companies, like you said, apart from shiny things, the culture is about reinforcing those that build new things. So I've been at places that the more microservices you build, the more likely you are to get promoted. But it didn't necessarily have to do anything with the business impact. SASHA: This is, to me, one of the dangers of SRE practice because SRE practice said toil is bad, so your general maintenance work, your work that you're doing day to day to keep the lights on and stuff. It's all bad. We want to automate it away, and we want to stop doing it. Or we want to minimize it. And then Google said 50% toil and 50% development of automating things and making things better. But I've seen companies that are like, oh, we are 20%, toil. Like, really? Do you think you can get away with 20% operations work?  And it is a problem on a couple of dimensions because one is I just don't have enough time. Because I can't automate a thing if I don't know how this thing works. If I don't know what breaks, or what's brittle, or what's not functioning correctly, I can't write code to automate it because we just don't have the context. So you need that toil to learn what your system does and when it's not working to be able to make things better.  And the second thing is what you said is like, if I'm only rewarding code, so if to get promoted I need to write new features, I have a horrible job. If I do toil, work like I do incident management 90% of my time, and this is what is actually making my company reliable, and then I don't get promoted because I didn't write code. ANA: Yeah. And we still see that a lot in the industry, even through a lot of these transformations that we've gone through. So I'm curious to see how is it that, as an industry, we could start approaching it a lot more head-on? I know some folks who are tracking more of those metrics of deployment productivity of employees. And I don't necessarily fall into the box that you should do that and see it progress. But I also haven't come up with another alternative. SASHA: Incentives are always dangerous. Designing your incentives is complicated. Honestly, there are a lot of companies that don't have any metrics at all, so people get promoted randomly or never. It's like, for smaller companies, there's a lot of my manager likes me kind of thing and whatever. And then you have big companies that are like, oh, you need to write about [chuckles] how your impact was good for the organization.  ADRIANA: [laughs] So true. SASHA: And it's just disheartening. Why is half my job proving that I'm valuable to you?  ANA: [laughs] SASHA: That shouldn't be the case. It sucks. So one of the things that we talk about at Ergonautic is you need a system of metrics. You don't need one metric, and you don't need a list of metrics that are disconnected. But you need to make sure that your metrics are kind of balancing each other. I'm not saying Microsoft was perfect, but at one point, I discovered that it was all about, like, oh, I implemented this new feature because people wanted to get promoted. And so it led to every department has their own testing tool because everybody wanted to get credit for implementing the new testing tool. They don't get credit for using the testing tool. They get credit for implementing the testing tool, so that's the incentive. So they tried to change it to like the three-part thing, so it's like your contribution as an individual, your contribution to your team and how did you help your team and the organization, and your contribution to leveraging your team and the organization. So kind of trying to remove this lone wolf, I'm going to solve all the problems by myself. And, again, this is never perfect, but it's maybe a little bit better than incentivizing every person to write a pamphlet about how they're solely responsible for all the reliability at Google or something. [laughs] ANA: I mean, it goes against hero culture.  ADRIANA: Yeah, it's true. ANA: Like, we're aware of the on-call hero. But at the end of the day, dev teams also have heroes like that where it's just like, land grab and try to do as much as they can, that amount of push the most amount of code, do as many reviews as possible. And it was like, but were you actually being a teammate and using features and thinking about removing tech debt? SASHA: Hog knowledge, right? So no one can do what I do because I'm the only one who understands how this shit works. And I'm just not going to share any of this information. I'm going to roll my eyes at everybody else and just say, "Oh, you just don't understand."  ANA: [laughs] SASHA: There's a lot of that, unfortunately, in tech in general just because they were kind of rewarded for being smart. And they're still like...I don't think it's pervasive, but there are still pockets of this 10X engineer, whatever.  ANA: Yes. SASHA: If you remember "The Phoenix Project," Brent is this guy who knows everything. And there, he's not toxic; he's just ended up with all the knowledge. He's a problem. Like, he's a problem because he's a bottleneck for the entire system. So you're a single point of failure, human. ANA: Yeah. [chuckles] ADRIANA: Yeah. And it's funny because I remember reading that book, and I'm like, oh my God, how do you know my life? [laughter] Like, not me as being the bottleneck, but I've met so many Brents in my life, and I'm like, this is wild. ANA: [laughs] That was definitely a great read.  SASHA: Yeah, I gave this talk years ago, and I said there was a single person of failure. I only gave it once. I never gave it again. And it's like, someone pinged me six months ago and was like, "Oh, this is a great talk. I related so much." I'm like, oh, nothing has changed. [laughter] ANA: I'm glad you saw my content, but oh no. [laughter] SASHA: No, it's cool. I mean, I'm glad you liked this talk six years later, but like, oh no. [laughter] ANA: I mean, I think we're always going to be seeing some of those things bottle up again, or it's like the same type of behavior comes up. Like, we don't learn from our past learnings, just as humans, until we fall all the way down deep. And it's like, someone else really points out there's a lot of Brents at their organization, and they can do something about it.  ADRIANA: I do feel like at least now we're, in general, on a kinder, gentler path where these things are, like, there's awareness being brought to a lot of these issues. I'm not saying that they're necessarily being solved. But I feel like the fact that we have conversations about these things to surface them, I feel like there's hope. You know what I mean? SASHA: Yeah. I mean, it helps, at least when people are aware and when people have the language to talk about certain things. Like, if you don't have the words or the idea is not generally accepted, it's really hard to explain what the problem is. Once there's some awareness, you can at least be like, okay, if we have this problem, here's a book on how is this a problem. ANA: [laughs] ADRIANA: Even when we're talking about things like psychological safety and SRE burnout and stuff, these are things that were just kind of accepted back in the day. And now it's like, no, we're having conversations about well-being. So that makes me happy. Because certainly when I started my career, this idea of like, oh, you can't push back because you got to earn your place in the company. And now I feel like we work in a more, I guess, accepting tech society where we speak out, and sometimes things get action, sometimes they don't. But at least I feel like people are more emboldened. SASHA: Yeah, maybe; I don't know. I don't feel like I see so much of a trend. It's more like I feel empowered to change the place of work. So either I try to change the organization, you know if you can't change your job. Like, there's one of two things, either I'm going to make things better, or I'm just going to leave.  And then the difference between a good organization to work at and a bad one are so vast. You don't want to end up showing up to work every day hating your life, and psychological safety is a huge part of it. And I think early in a career; you just don't have as many choices. You feel like if you leave after a year, that's going to look bad on your resume and this and that, whatever. And so I think that's some tenure freedom that comes with experience. ADRIANA: Yeah, yeah, that's true. That's true.  ANA: [laughs] ADRIANA: I think the more established you get in your career, you just give less fucks [laughter] and care more about your own mental health and what you'll tolerate. ANA: Even just being able to recognize what's terrible behavior. Like, this is not supposed to be long work hours for this long period of time, not just like here and there for finishing up a project. Those things end up catching on. ADRIANA: Yeah, that's true. SASHA: I mean, also in your 20s, sometimes you don't mind working a lot doing this kind of stuff. And then you kind of get older, and it's either you get more responsibilities in other ways, or you're just like, no. ANA: [laughs] ADRIANA: You're just like, no, I'm done. SASHA: You're just like, this is the boundary. [laughter] ANA: The wall goes up here. This is my schedule. If you don't like it, I'm leaving the organization. [laughs] ADRIANA: Totally. ANA: I wanted to go back to some of the stuff that we were kind of talking about earlier, that we were joking around about all these buzzwords that kind of have come out of, like the DevOps movement, the SRE movement, and now the platform movement. With these buzzwords, what are some of those most cringe-worthy terms, in your opinion, that you keep seeing? SASHA: I don't know about cringe-worthy. My brain isn't serving me anything specific. But I think we talk a lot about this continuous buzzword creation, so I'll give you an example. A lot of people think they do Agile, and they actually don't do Agile. But you can't come in and be like, "I'm going to teach you Agile," because they're like, "No, no, we're already doing this." "How are you doing it?" ANA: [laughs] SASHA: "Well, we have standup every day." [laughter] Okay. And the thing is there's no perfect methodology. And it depends what kind of organization you have and the size and complexity, and you kind of need to be able to adopt the practices to what matters to you. But also, I always say there's no perfect way to develop software, but there are bad ways. Like, certain things you're like, oh yeah, you should not do that. [laughs] And then there's no...I don't believe in there's a gospel, and you should only do one thing only one way. But there are always ways to improve. So if you're familiar with the Theory of Constraints, you have these bottlenecks in your system, and usually, improving the flow through the bottleneck improves your system. But also, your bottlenecks sometimes shift. And so we had a bottleneck here, and now this has been relieved, so now you have another bottleneck there. And there's always something that can be better. I think the problem with buzzwords is this continuous renaming. So it's like, oh, we're SRE now. And again, people say, oh, whatever SRE transformation failed. And are you doing SRE?  ANA: [laughs] SASHA: Show me what you're doing. Have you tried this? Have you tried reading the book? [laughs] ANA: Even when you read the book, they want to implement exactly how Google implemented it. And it's like, you're not even close to the size of Google, or you don't have that talent. You need to be more scrappy or just think differently. SASHA: Well, so the other thing is the Google SRE was written by a particular subset of people at a particular point in time. It doesn't mean that all of Google does exactly this thing all of the time. And the other thing is, yes, the complexity and size and number of engineers, but the other thing that a lot of people don't have is this opinionated platform. Google actually says, "We're going to run the platform for you, and you're going to use it." And I see a lot of organizations start working on a platform, but then it doesn't get adopted or something like that. Or the developers are allowed, for instance, to develop in any language they want. It's like, so do you think that developers at Google are allowed to develop in any language they decided? That's not the case.  ANA: [laughs] SASHA: Like, the operations burden increases exponentially if you keep adopting new technologies, and it's kind of like the wild west of what could potentially be happening out there. And I'm not advocating necessarily for, hey, limit everything, but there's this concept of enabling constraints.  ADRIANA: Can we dig a little bit into starting your own business? I think you alluded to it a bit as to what motivated you. But how did it come about? How has the experience been? SASHA: So it's interesting; I've been thinking about starting a startup for like a decade. I've never had a concrete idea of I want to do this or that. It just was an idea that I want to start my own company. I want to try how it feels like. There was a combination of things. I actually ended up working for bigger and bigger companies, and I'm like, oh gosh. [laughter] And it's hard to go try something on your own when people keep offering you promotions and salary raises, and you're like, this is going pretty well. Why would I go try something new and potentially make no money?  ANA: [laughs] SASHA: And then I think the timing was just right. So I got to director level, and I was feeling like I don't want to be a VP at a big company. This is not the career path that's exciting to me, at least not right now. Maybe sometime down the road, but not right now. And the farther you get up the management chain, the more it's about politics and less about, like, I don't want to dismiss it. Playing political games is kind of fun too. It's got its own kind of excitement, but it just kind of wasn't what I wanted. And then my co-founders just came around, and they were like, "Hey, we're starting this thing." And I was like, "I want in." And it's like, "Are you sure?"  ANA: [laughs] SASHA: "Yes, please. Can I please join you? This sounds amazing." So this is an adventure. Like, for months, I was like, I don't know if this is going to go terribly wrong or if it's going to be terribly right. It was kind of trepidation. It's so new. I always had a paycheck. I didn't have to worry about how do I do 401K? I don't know. ANA: [laughs] SASHA: How do I do my taxes? Everything is so, like, I'm doing research on the internet. How do you do this, and how do you do that? And it's like, in addition to you're trying to actually work on a thing. So it's like a totally new world. But it's been really exciting because we're building our own thing. We're defining our strategy, and we're defining what we're delivering and how we want to grow, and how we want to position ourselves, a very new world for me. I worked for a tiny startup before, but I've never owned one. So it's like a whole new world of possibilities here. ANA: [laughs] That's so exciting. You get to put out this little project, feed it, and see it grow into this hopefully bigger organization that you get to be proud of. SASHA: Yeah, yeah. And to me, I define success like if I learned a lot of new things and made some money; this is already a success. But if we actually get to do what we want, which is to grow the organization, scale it out, and hire a bunch of people, be really good at the type of consulting that we're delivering so we can actually help a lot of folks out there that will be just fantastic. ANA: [laughs] ADRIANA: That's awesome. And I do feel like you guys have a really cool little niche that you've hit up because there are so many organizations. I've worked for many large organizations, and it's like, yeah, they'll do their Agile transformation, their DevOps transformation, and then at some point, it tends to stall. So coming in and rescuing or righting the course, I think, is really valuable because we know the true power of what these kinds of transformations can give us. So giving those organizations the opportunity to really leverage those superpowers I think is awesome. SASHA: Yeah. And I think there's kind of a couple of things that are different in being an external consultant, so one is I've sort of consulted in various ways for different organizations, but I always had, like, I have to sell the software, or I have to sell this thing. I have to do more of this or that, and you're beholden to a vendor in some way. So like, yes, you're giving the best advice you can give, but you cannot say, "Oh, you know what? You don't really need Kubernetes," because your freaking company sells Kubernetes. [laughter]  And I'm completely free. I can recommend you whatever software I want or no software at all. I can say like, "You should start doing this. You should stop doing this." There are plenty of customers out there. We don't have an ulterior motive of like, oh, we want to sell you more consulting because it's not this kind of business. So it's a very freeing experience so far. And it's really really interesting to be able to talk to people really honestly about their experiences.  And then the other thing that's really interesting is I just get to sort of come in from the outside. So when you're internal to an organization, for instance, I cannot come to someone in my own company and be like, "You're not actually doing SRE. You suck. Just stop it. This is horrible." ADRIANA: [laughs] SASHA: Because it's like, people are doing the best they freaking can. And I cannot come at someone and be like, you know what? You should really hire someone's prior SRE experience instead of trying to onboard based on the internet. ANA: [laughs] SASHA: Because sometimes it is. Andrew has this metaphor of learning a new language. You cannot learn a new language without having a person who speaks that language. You need that ability, that expertise, and feedback. In theory, maybe you could learn the Latin language without any...but this is completely different. If I can be immersed in a situation where people are actually skilled at a thing I'm trying to learn, my learning experience is so much faster and so much better. So that kind of thing, like, sometimes you need to bring someone in who's going to be able to set the tone for the team to be able to do something.  So my point is, you know, I don't know if this is the best phrasing in the world, but the ability to say words and, of course, you kind of have to moderate. And you have to be able to message in a way that people can kind of relate to instead of being like, oh, everything sucks. You have to stop. But at least I'm not beholden to, like, oh, if I yell at these people, that's going to reduce my power in the organization. And then tomorrow, they're going to backstab me or something like that. ANA: [laughs] It does make a difference. And, I mean, a lot of it too is that coming from someone who's done it before or teaching them by practicing as well, like, going hand in hand and guiding them and walking with them. SASHA: Yeah, expertise matters a lot, and tools matter too. But it's the right tool for the job, not the shiniest new tool that you just saw. ANA: [laughs] ADRIANA: Totally. And as a quick follow-up, when you're coming in and basically telling teams like, "Hey, you're not doing this the right way," how do you approach it without them getting offended basically so that they're more open to the change? You're hired by a company, but it doesn't necessarily mean that the teams that you're working with are welcoming you with open arms. So how do you approach that? SASHA: So, for one, I think a lot of times, people are just happy to tell you. A lot of times, people know what the problem is, but they have no power to change things. So a lot of times, you just arrive, and people are like, "This sucks, and this is why," And they're already correct, but they just don't know how to fix it. And then the other option is you take them on a journey. So you expose them to ideas and kind of make it their ideas. Or you teach them gradually, like, hey, have you thought about this? And then that requires some continuous conversation.  But then basically, by the end of it, they're like, it's not a report that's coming from me, and I said you should do blah, but it's more of a collaborative effort where we thought together with you and everybody kind of thinks that this intervention is a good idea. And then the other thing is so when we do the report, we kind of list a lot of interventions, and we kind of list, like, okay, we think if you do these three things, it's going to be really, really powerful. And then there are those small fixes in different teams that you could do that could improve their life and whatever.  And so you can choose your adventure. If you don't want to do the highest recommended thing, but you think if you fix this little problem, it's going to be better, you can do that. But ideally, you know, we haven't been in business for too long. So I can't tell you how many of the reports are like; I can see that people are actually taking it to heart. But I think the goal is to actually have folks implement it and have their lives be better.  So that's the other thing is people are scared a lot of consulting companies because consulting companies come and do this big report that a lot of times is like, let's fire all these people, or let's replace all this software, you know, some kind of big things. No, you really should be better a little bit. You're not going to become an Olympic runner. But you could run a 5K after three months of training. That's kind of where we want to get you. ANA: [laughs] That makes perfect sense. And that's a really good way to start thinking of putting little chunks of something new you're trying to do into your practice and having your team work towards it. SASHA: Yeah, because the other thing we think about a lot is behavior change. Just mandating that people do something different isn't going to necessarily work. So how do you actually take people on this journey of learning a new skill and doing the thing? And ideally, there's automatically a reward in it. So I'm reducing friction. I'm reducing toil. I'm reducing the shitty part of your job, so you already want to do it because it's better for you. ANA: That's very true. I mean, it's that incentive of life will be better after this, or once we finish this transformation portion, we get to have an off-site. We got to have perf reviews and bonuses or things like it.  SASHA: Yep.  ANA: As we're getting ready to wrap up the episode, is there anything that's exciting you right now about the work that you're doing? SASHA: Yeah. So I think we're learning a lot of new things, and so one of my goals is...so right now, this is our expertise. So we have a few people who have done this for a decade or three. And they can come in, and they can give you advice because they've done this before. The thing that excites me the most is how do I write it down sufficiently, like what we do and how we do it that I can have a new person come in and we can teach people how to? So that's the exciting part is kind of getting knowledge out of our heads and onto paper. [laughs] ANA: Spreading knowledge is definitely hard. And it's always nice to be able to have those conversations with people, like, how did you train other engineers to do this type of activity? Or how is it that you learn from incidents? How is it that you pass on deep knowledge of your architecture to someone else or even just the entire DevOps movement? [chuckles] SASHA: Yep.  ANA: Well, thank you very much, Sasha, for joining us in today's podcast.  Don't forget to subscribe and give us a shout-out on all social media for oncallmemaybe. Be sure to check out our show notes on oncallmemaybe.com for additional resources and to connect with us and our guests. For On-Call Me Maybe, we're your hosts, Ana Margarita Medina... ADRIANA: And Adriana Villela signing off with... SASHA: Sasha Rosenbaum. Peace, love, and code. ADRIANA: Whoo!  ANA: [laughs]

About the guest: Jewel has been shitposting and sadtweeting her way to a comfortable tech career for over a decade in community and consumer product engineering as a software engineer, manager, and all-the-hats start-upper.  She hasn't accidentally taken down production in a whole month, mostly thanks to PTO. Find our guest on: Twitter LinkedIn GitHub Instagram Mastodon Find us on: On-Call Me Maybe Podcast Twitter On-Call Me Maybe Podcast LinkedIn Page On-Call Me Maybe Podcast Mastodon On-Call Me Maybe Podcast Instagram On-Call Me Maybe TikTok On Call Me Maybe Podcast YouTube Channel Adriana’s Twitter Adriana’s Mastodon Adriana’s LinkedIn Adriana’s Instagram Ana’s Twitter Ana's Mastodon Ana’s LinkedIn Ana's Instagram Show Links: Reddit Ren Faire (Renaissance Fair) Commodore 64 Jumpman (Video Game) TI-82 (Calculator) TI-83 (Calculator) Epic (Medical Records) Epic Games Workday Sonic Foundry Imgur mipsytipsy (Charity Majors on Twitter) The Engineer/Manager Pendulum - charity.wtf Engineering Management: The Pendulum or the Ladder - charity.wtf Pop Psychology Transcript: ADRIANA: Hey, y'all. Welcome to On-Call Me Maybe, the podcast about DevOps, SRE, observability principles, on-call, and everything in between. I am your host, Adriana Villela, with my awesome co-host...  ANA: Ana Margarita Medina. ADRIANA: And today, we have with us Jewel Darger-Sacher, who is a Senior Full-Stack IT Engineer at Reddit. Welcome, Jewel. JEWEL: Thank you. I'm delighted to join you all today. ADRIANA: We are super stoked to have you with us. Now, our first question that we always ask our guests is what are you drinking today? JEWEL: So today I brought some fancy tea. I picked up some raspberry chai from Ren Faire a couple of years ago, and it's treating me great today. ADRIANA: Awesome. How about you, Ana? What are you drinking? ANA: That sounds very yummy, and I kind of wish I had some tea. I went with my LaCroix peach-pear sparkling water. Sometimes you just need a little bit of flavor. But I need sparkling water to do podcasts sometimes. It's just like that nice, crisp of staying awake. What about you, Adriana? ADRIANA: I am finishing off some green tea, and I've also got a glass of water. ANA: Extra hydration. ADRIANA: Extra hydration. JEWEL: Always staying hydrated. ADRIANA: That's right. So I guess first things first, Jewel, we always love to hear how our guests got started into tech. It's always really cool to hear different people's paths to their current career. So, what's your story? JEWEL: I grew up with a keyboard in my hands. My parents actually worked in IT and computer science from their graduate degrees. I was the third child. And they bought a Commodore 64 for the house to play games and do word processing. And I was the baby. I'm the third daughter. So I got to play a lot of Jumpman and little machine games as a kid before I really knew anything.  And I remember they had just a big button or a big sticker on the go button on the keyboard for me because that was all I could push when I was a baby. [laughter] So my parents kept that up over the years. They gave us a good widespread diet of arts, and music, and math, and tech, and literature. So we always just had computers lying around in the house.'90s kids will remember the TI-82, the TI-83. I think they're still getting used.  ADRIANA: Yeah, yeah. JEWEL: Yeah. So I spent a lot of time in high school just manually hand-typing little games into the calculators, so I could goof off during class. ANA: [laughs] ADRIANA: Oh my God, that's awesome.  JEWEL: Yeah. And then there's the one time where my chemistry professor walks over, and he's like, "Excuse me, miss, you seem to be goofing off. What are you doing?" And I went, "Oh, I'm programming a visualizer for the orbit of electrons around this molecule," [laughter] using matrix math that my dad had taught me because he's, as I mentioned, a math nerd. So yeah, the instructor was just like, "[sighs] carry on." ANA: [laughs] ADRIANA: I mean, how would you respond to that? Like, you've out-nerded the instructor. [laughs] JEWEL: Yeah. So I definitely come from a nerd background, and then I did actually -- ANA: [laughs] ADRIANA: That's awesome.  JEWEL: Yeah. And then I actually did get a real Bachelor's degree in computer science. But I kind of mixed it up a little bit where I got a Bachelor of Arts in Computer Science. ANA: Nice. JEWEL: With a double major in technical theater and a minor in Spanish studies. And people thought I was a music major. ANA: [laughs] ADRIANA: That is awesome. That's so cool, so cool.  ANA: So many different buildings [laughs] to be in all the time.  ADRIANA: Wow. That's a lot of hats. [laughs] JEWEL: It's a lot of hats and a lot of keys. I also worked in the AV department, so I would have keys to a lot of campus.  ANA: [laughs] JEWEL: It was really funny when the campus security guy called me once or twice, and I'm just like, "I'm sorry, I work here. Like, I have more keys than you do." [laughter] And they were not impressed. But yeah, so that was my...my informal training as a kid was just mess around with computers, do math and science stuff, do a lot of other interesting extracurriculars, and then went to college. And then, after college, I got into the industry. I worked at Epic Medical Records, not Epic Games, doing internal tooling there where I worked on basically the internal employee directory and how to request PTO because this was 2009, and we didn't have Workday at that point.  ANA: [laughs] JEWEL: We built too much stuff in-house. And then, I worked at Sonic Foundry after I got tired of Epic and mostly the commute. It was an hour commute in the snow sometimes, which was less fun. But Sonic Foundry was a quick skateboard ride away from my apartment. And then I left Wisconsin for California when one of my friends who I had been working on a startup with, said, "You can either move to Rhode Island, and we can get acqui-hired, or we can move to Silicon Valley, decline the acqui-hire. I have some other jobs lined up." And we said, "California sounds nice." ANA: [laughs] JEWEL: So we moved to Mountain View, and we worked there for a couple of months before we got laid off, which was fine. I didn't like the job anyway. And then, I got a job at Imgur for a year. And then I've been working at Reddit for the past six and a half years. So yeah, that's been my...I've just kind of floated around tech a bit, just kind of poking my nose in a lot of things and trying to make it multidisciplinary. ANA: [laughs] ADRIANA: That is so cool. ANA: It's always fun when you end up getting a chance to bring your hobbies into work and in and out and try to figure out what makes the most sense. I know when I first got a chance to meet you, I was just like, wait, how did you even study Spanish and something theater-related and technology in your year? And at that moment, you were a manager at Reddit.  It was just one of those things that I was like, okay, acknowledging the different career paths that folks can come in with, which was actually really cool. But now that we have you here as an IC software engineer, I also kind of wanted to poke at it. What was your transition from management to IC like? JEWEL: It was an interesting transition to go back from being a manager to IC. And I'd always kind of planned this where I remember reading mipsytipsy's pendulum career path blog posts. So Charity had written several years ago about the benefits of working as an IC for a while and then switching over to engineering management and then switching back to an IC path as, you know, more experience and more understanding of how does management work? Why does management work? Like, what do the managers need? Like, why are they asking me these things that I previously thought were a name? And now I come back from it, and I'm like, oh yeah, I get this. ANA: [laughs] JEWEL: You're asking for this because your directors are asking for it, or because we have this regulation to deal with, or this policy, or this process. And I also understand why you build processes because, as a manager, you usually don't have time to deal with stuff. So you write it down, and you ask somebody else to follow the instructions. Or, more realistically, you ask somebody else to write down the instructions for a third person to follow.  ANA: [laughs] JEWEL: So yeah, it's been fun going back from manager to IC and just having a lot of empathy for my manager and taking a lot of work off of his plate, especially logistics and project management that before I wouldn't care to deal with it. My manager would be like, "Could you please fill out your paperwork?" Now I understand why he's asking me to fill out the paperwork because he wants us to keep getting paid. ADRIANA: Yeah, it's funny because, as a manager, it's so frustrating to get people to fill out timesheets or fill out employee surveys and stuff like that. And then, as an IC, if you haven't been on that side of things, it can be a little bit difficult to relate to just how frustrating it is to try to like herd cats [laughs] because that's what it feels like a lot of the time as a manager. I definitely agree with you on the empathy.  ANA: [laughs] ADRIANA: I definitely find that nowadays when my manager asks to do some admin thing, I'm like, oh yeah, I remember what it was like to try to get people to fill stuff out and how frustrating it is to chase people down. And it's totally out of your control because, I mean, they have to do it, not you, but you're still accountable for making sure that that stuff gets done.  JEWEL: Right. It's like, I see the benefits of, like, oh, if I do a little bit of the admin work, they can help me out with a lot of other things that they're much better suited for. Like, my manager is really good at schmoozing, and he has a good sensibility for, like, here's who to talk to, here's who you don't talk to, or like, here's the person you talk to first. They will help you prepare for who to talk to second because that person's time is hard to schedule, or they have certain interests they want to zero in on.  And it's like a lot of stuff that I just don't have experience in. I'm like, oh, you've been around this block. Let me just watch and learn and take coaching. And then he also does the nice thing of career growth for me and puts me in those situations and then back channels it to me about, like, you can lay off this topic. You can move on to that. Or, like, hey, you're doing a really good job. Keep it up. ANA: That's super supportive. [laughs]  JEWEL: Yes, very supportive. ADRIANA: I was going to say you have a unicorn manager. [laughs] JEWEL: Yeah, he's pretty great. He even asked me in one on, one he's like, "Hey, I know that you've got this big three-month recovery time for surgery coming up. Are you planning that in Q3 or Q4 or what?" And I was like, "Let me get back to you in two weeks after I talk to the surgeon." So that's what I'm dealing with today is going back and forth, back and forth with the surgeon, to nail down times so that I can take off three months from work. My manager is here for it, and I'm like, oh, great, cool. This is awesome. ADRIANA: That's awesome.  JEWEL: And his request was like, "Please give me two or three months' warning so that I can schedule our projects around this." ADRIANA: Yeah, for sure, for sure. You mentioned going from manager to IC. Have you done the jump back and forth a couple of times, or is this your...or have you just done the jump once from manager to IC? Have you gone manager to IC, IC to manager? JEWEL: In terms of titles, I have been an IC software engineer for years and years and years. And then, I took on a management role for three years, two and a half years, and then switched back to IC. But I've also been doing lead roles in other spheres, like for ERGs. I've helped found and lead and then train up other leads and step back and get them to do the work for a bunch of ERGs and for various little projects.  One of the nice things that we do at my job is like a one-week hack week every quarter. So even though I've come back from being a full-time manager to an IC, I have this time where I can put together a small squad for a week or two and do engineering manager work again for them or act as a product manager or act as a project manager and delegate out other stuff. Mostly it's to ask people to step up and pick up who wants to be the project manager for this or who wants to do this and that. So I get to jump around to a bunch of different leadership roles, even without the manager title now.  ADRIANA: Oh, awesome. Lots of fluidity, then. JEWEL: Yeah. It's fun, especially during this one week off where it's like, go try out partnering with people in other teams, go try out other roles, go just do some project you've been wanting to take care of. So I end up kind of matching up around my ERG goals as well. Like, one of the projects that I worked on last quarter and I'm working on this quarter is when you change your name...because I changed my name legally and socially last year.  So, in addition to changing your government ID, there are all these places where you're identified by name or by username. In all these systems, what's the process for getting those fixed, like, for updating those? Where does it kind of break? Where does somebody need to manually take on work versus where can you just do it in one central point?  So one of my projects was, as I went through that, I went and logged all my actions that I was doing, like, I updated this system, or I noticed that this system was broken, just log it. And then, after three months of that, I got together a group for this hack week and said, "Hey, I need people to help me get this knowledge together, get it usable, identify more gaps, and turn it into a guidebook, like a runbook for other on-calls who are tasked with processing a name change of one of their fellow employees," because this happens all the time. People are changing their names for weddings, and divorces, and transitions, and just notable life events. And it's like, okay, we should streamline our systems where they're not streamlined because there are protocols for this. But there are a lot of systems that are on older protocols or aren't doing it. So we have to do manual work on top of that. So it's like, all right, well, let's write a runbook. That's what we know how to do, write a bunch of runbooks. ANA: [laughs] It's nice that you get a chance to bring some of that mentality of, like, you do it once, and now let's actually document it the first time so that we actually don't have to deal with this on a constant manner as it continues to come up. That mindset of the SRE that we kind of get to walk life with, I think, is a very niche space of glasses to wear and pattern-recognize, or just kind of want to standardize, automate, delegate? JEWEL: Mm-hmm. Yeah, it's very much where can we make the computer take on the work? And where do we need to have our own checklists to take care of it ourselves? Or can we even find middle ground of you file a helpdesk ticket, and the automation is not go apply the change to all the system? The automation is go create more helpdesk tickets, like, have one central helpdesk ticket, that is, I'm changing my name. And that can fan out to the accounting department and the infra department, and SREs, who are people who are responsible for different systems. We have a team that is specifically responsible for the feature flagging and experiment framework. They have their own little data store that has data mapped by your email address to you. So if you change your email address, you need to go change data in those data stores. It's like, cool, let's just fan out a dozen tickets, and we just know where they need to go. And we just automate creating helpdesk tickets more easily than making somebody walk through a checklist on a Wiki page. ANA: Make it easier for the user and follow the processes as needed. It's beautiful. [laughs] JEWEL: Yeah, exactly. Like, it's hard enough to deal with a name change in general. It's like, yeah, let's just make it an easy process for everybody. ADRIANA: Otherwise, when these things become so complicated, it becomes incredibly off-putting. So in some cases, it makes you want to just give up, or you invest a lot of time in this process and it kind of drives you crazy. [chuckles] JEWEL: Yeah, a lot of time, a lot of money. I talk to people who are just like, "You know what? I don't want to bother changing my name when I get married because it's so much logistical overhead." ANA: And, I mean, I think it even adds on more to the overload that kind of comes with it, like, something that should be simple. Like, why should the employee be suffering when using all these tools in order to do something so basic as, like, legally, this is my preferred name? Legally, it says that it is. Why can't my job kind of catch up? And that non-inclusion space and what that means for the person that they kind of have to take on of like suffering and feeling like they can't be themselves, or just extreme frustration that leads to a lot of other mental health aspects. JEWEL: Yeah. And it really puts a light on where has there been investment in this? Banks have it moderately streamlined, not that great. The government, like, driver's license that's streamlined. Social Security that's pretty streamlined. But updating your gym membership, no, they don't care. ANA: [laughs] JEWEL: They're not dealing with financial regulations as much. And one of the things in particular that I saw is our benefits system; our benefits reimbursement system broke, where I couldn't log in. I had to go get a separate username and address. I couldn't login with our one-click login because I had set one name in our internal systems, but I hadn't changed my legal name yet. So the benefits system was still working off my legal name, and they struggled with that.  ADRIANA: Oh no. JEWEL: So that was just like, cool, I have to jump through even more hoops to get money to pay off for all this other stuff going on in my life. Fortunately, I did notice after I changed my name at social security that did percolate through a surprising amount of systems automatically. It's like, okay, if I did start with the blessed path, in some ways, my life would be a lot easier as opposed to I was impatient for various reasons and did things a little bit backwards.  And I'm like, I feel like this is a lesson I keep learning a lot at work too of like, when you're starting up a new service, and you're like, you want to build a new application, and you want to get it out the door. We have at work this great system for...you go talk to the SREs and the infra folks. And they talk you through what kind of load do you expect? And what are your failure points? And how are you going to recover when they fail?  And what are the dependencies that your app is sitting on? What's going to be dependent on your app? And when they go down, how is your system going to recover? And how are the other systems going to recover from your app going down? So they just walk you through this whole system of making sure you're going to have this nice, stable system to work with, and you can have a nice, smooth launch.  Because when you're working with a daily user base of...I try to avoid looking at the numbers but lots, there's a lot of traffic going through. And if you want to spin up a new service, you have to target for stability from the start. There have been some times where I just went ahead and built stuff from memory, and I did not do the checklists, or like, I didn't go and talk to my SRE partners. And I went, "I want to go launch this thing." And they went, "Okay, are you sure? ANA: [laughs]   JEWEL: Are you ready? Are you planning to stay up all night to keep fixing this because we're going to have to stay up all night fixing it with you. We would rather get sleep because sleep is good. You like sleep too. You've done on-call." I'm like, "Oh, yes. Yes, that's a good point. I should follow through with my checklist." And this is why I appreciate my SRE partners who then go, "Okay, we'll fast-track, walking you through the checklist because you have an urgent deadline. Buuut... ANA: [laughs] JEWEL: You need to go through the checklist, or you're going to be hurting more than if you just push your deadline another week or two." So sometimes it's easier to take the blessed path, take, you know, somebody built the system up, and it's like, okay, fine, I'll follow the system. It'll make my life easier in the end, I guessss, fine. ANA: The checklist. ADRIANA: But also having people to hold you...being, I guess, a voice of reason helps a lot.  JEWEL: Yeah, that externalized voice of reason, especially because I get too close to my project, or I get too invested in just shipping it. And they're like, "Yeah, you can ship this. It'll be easier if you do this very reasonable thing we've asked." ANA: [laughs] It's like the checklist got created for a reason. Like, let's continue preaching to standardize it. JEWEL: Right. All of these things on this checklist are because somebody got hurt at some point or somebody got hurt repeatedly, and we would like to not repeat those same mistakes. We would like to make interesting and new mistakes. ANA: [laughs] ADRIANA: But it's interesting, too, because it can be a double-edged sword. Because, I mean, it sounds like your checklists are actually up to date and are useful. And then you've got the other scenario where you've got these legacy checklists that nobody bothers to update. And then you don't even know why these items are in the checklist, and then you end up screwing yourself over for that. So it sounds like you have the happy path of checklists where the checklists are up to date, are doing what they ought to be doing. JEWEL: Yeah, it's very nice to be in a company where we are trying to keep everything fresh, especially with...there's been a lot of just change under the surface and the inner workings of the platform in the last couple of years, so it's a lot of, well, we used to use this process, but it doesn't even apply anymore. Let's look at it and see what we can pull out and reuse. But sometimes it's like, yeah, you've forgotten what this is useful for? Well, keep an eye out, and if it's important, it'll pop up again, hopefully, probably.  ANA: [laughs] JEWEL: But it is that constant like, yeah, just be tweaking your processes. Go back and look at what you're doing every couple of weeks or every couple of months and just tweak stuff. And I actually have a reminder. In my team's core project, in the README, I have a checklist of, like, did you add screenshots? Did you add verification instructions? Did you add post-deploy monitoring instructions? And then there's one more item on the checklist that says, "Have you noticed you have been going off the checklist a lot recently? Like, are there things in the checklist that you're just constantly ignoring or things you keep adding? Go change the checklist. You have the power of changing your own checklist." ANA: That's a nice little reminder to add for your future self and that mindset. JEWEL: Yeah, because it's like, I remember...I'm looking at a photo on my wall where I astonished one of my friends because she...I was just showing off like some art. And I have this black and white portrait of just a line drawing of a woman. And she said, "Oh, it'd be really cute if you had a copy of that with blue hair like you do." And I said, "Oh, that would be cute." So I took it off my wall, and I took a marker, and I drew on it. And she was astonished. ADRIANA: [laughs] JEWEL: She's like, "Wait, you can just draw on that?" I'm like, "This is a $20 painting from Target. I don't give a shit.  ANA: [laughs] JEWEL: Also, it's got glass, and this is a dry-erase marker. So when I dye my hair again, I can fix the color. It takes 30 seconds." [laughter] It's like, yeah, just reminding people of, like, we're not stuck with the tools we're given. We have so much power over the tools that we use and the ways that we use our tool. And it's just fun to just remind people of, like, you can tweak your stuff. Like, it's a little frustrating to have to constantly be tweaking your tools. But it's nice to have the option of, like, it's been two or three times this wasn't working. Let's do something different. ADRIANA: Yes, and being given the space to do that as well. I think one of the things that I've benefited from most in my career is being in places where when something's not working, being given the space to take a step back and rethink how I'm doing stuff versus places where they're like, oh, this isn't working, you suck. But you must continue doing it the crappy, shitty way. JEWEL: Yeah, I definitely appreciate having managers and peers who will remind each other of, like, it looks like you struggled a couple of times. Do you want to host a post-mortem? Do you want to maybe make some tweaks to the system? And getting that messaging from our VP of infrastructure is a big proponent of monitor yourself, monitor how you're working.  And when we start doing a post-mortem discussion, there's the preamble, which we will repeat even if it is a bunch of people who are experienced. But the preamble is we work in complex systems with a lot of people and a lot of inputs, and things constantly changing. Things break constantly. Let's talk about it. Like, this is the blameless post-mortem style of you might have done something, but we understand that you are trying to do your best.  And if we see a trend of somebody not trying to do their best, then it's like, that's a different discussion. But we're just assuming that good intent off the start of we're trying to do our best in a complex and challenging environment. How can we support ourselves and each other? And just having that throughout the company it's so nice, and just having lots of places where we can go ask questions like different channels for different topics.  Or every week or two, there'll be a meeting of front-end people, or back-end, or machine learning where it's like, yeah, let's just keep in touch with each other and figure out how to help each other constantly. And that's when people are interviewing, or people just joined the company, I'm like, how do you do success? You help each other out. ANA: Humans first. I mean, I think that acknowledgment you share is like, if more organizations were actually starting to conduct their meetings around incidents in that manner, they actually might be able to understand their systems more in a faster way than when they're sitting there pointing fingers of like, so and so did this but this didn't do that.  Versus focusing on why is it that such person did this action? Like, because the UI was easy to use? Is it because of prior knowledge, or is it just because that was the first place that you thought of checking in your tooling in order to see if there was an issue going on? Like, asking those questions actually to protect one another a lot more. JEWEL: Yeah, asking the questions. And it's like, yeah, you find out so much more. And you can find so many opportunities that are non-linear. It's like, how do you think outside the box? We just ask a lot of questions that lead you outside of the box. And then we're like, oh, is this the first thing you saw on your tooling? Okay, we can fix the tooling. Or like, oh, you never got trained in this new tool? Was there any marketing around the new tool? Okay, let's do a little round of marketing around this new tool that's more useful for you. ANA: They sent you to the wrong dashboard? What link did you go through? And then you start realizing that you have a whole bunch of random dead links somewhere else. JEWEL: Yeah. Or, like, we had this recently where we moved one of our Wikis and didn't realize that we wouldn't get redirects. So now I do have a bunch of Wiki links that are sitting around that I need to go fix up. So occasionally, people will be like, "Hey, your link's broken." I'm like, "Oops, thanks.  ADRIANA: [laughs] JEWEL: I'll go fix that. Where did you find it? Are there more nearby that I can go fix in the process?" Yeah, asking questions is..., and I'm finding this more and more, especially as I'm growing older and getting better at personal relationships outside of work, that asking questions gets you really far and assuming that people are trying to do their best.  ADRIANA: So true. JEWEL: And if it's like, oh, I feel like somebody is not doing well by me. Then it's like, well, let me ask them, like, do you know what I think is good? What do you think is good? Do we need to negotiate a little bit, or do we have some fundamental mismatches? Or where can we find common ground here and take care of each other outside of work?  And it's very much the same style of questioning that we use in a post-mortem question or a pre-mortem of, how is this going to go wrong? What happened? And let's just talk through it and not be blaming each other. Let's just take note of things and figure out interesting follow-ups.  You just have to change the framing and the phrasing a little bit where it's like, if I'm talking to my partner, like a romantic partner, I'm not going to be phrasing stuff in like, what's our OKRs for this weekend? [laughter] I'm going to be saying, "Hey, what do you want to get done this weekend? What do you want to get up to?" And it's like, "Oh, you want to go out and have dinner and get a nice dinner? Okay, we can work towards that." That's our OKR of, like, objective: have a nice dinner. Key result: we get home, and we feel fat and happy ANA: [laughs] ADRIANA: Bringing SRE into real life. ANA: And, I mean, I think even as you were mentioning earlier, that part of asking questions and trying to understand there is that part of, like, where are they coming from? Why is it that they're thinking this? Why is it that this behavior happened? That we get to have that context in humans around us and relationships around us, that we get to ask more questions to try to understand someone's behavior in the same way that we try to understand the system's behavior. JEWEL: Yeah, because humans are big systems, and systems are representations of humans at scale.  ADRIANA: Absolutely. ANA: [laughs] The only big difference is that humans just have this other part of systems call called feelings and emotions. [laughter] ADRIANA: Throws a wrench into life. ANA: So... JEWEL: That's one of the really fun things I've been chatting with in therapy in the last couple of years. It's like, okay, what are the systems that I can use around processing emotions where, like...I heard this phrase, like, feel your emotions. And I'm just like, how am I supposed to feel my emotions? And then, thank you, pop psychology on Instagram. I saw a little diagram that's like, here's how emotions manifest physiologically.  Like, when you're feeling anger, you'll feel this literal tightness in your throat or this heat in your chest. And it's like, oh, I recognize all these references from theater. When you feel that gorge in your throat, like the tightness, it's like, no, you're literally feeling tightness, and that's part of your brain, your body signaling that. And it's like, oh, this is my dashboard of, like, let me pay attention to that. ANA: [laughs] ADRIANA: Your body sends you alerts when things are not going well. JEWEL: Right. And that's when it's like, oh, the yoga instructors are like practice mindfulness. I'm like, mindfulness around what? [laughter] Oh, that my body is sending me these alerts that I need to sit up and eat something. Why am I feeling angry right now? Oh, because I need to go eat something. I need to go take a nap. ANA: We got shipped with tooling, no runbooks, no Wiki. We're out to figure it out for ourselves because it's that, where it's like, feel your feelings. And you're just like, I feel sad, and I'm going to go on about it. But you don't really necessarily understand where in your body that actually shows up.  I've also been following some Instagram holistic psychologist accounts that were talking about that where it's like, where is it that anxiety pit in your stomach actually falls and where that stems from. You actually not being able to talk about those feelings, and then your body internalizes that processing. So it's kind of interesting on that human complex system mindset. JEWEL: Yeah. And, like, we do have some of these runbooks. This is where it's like you get graduate degrees in therapy and in psychology, and it's like, oh, they're the ones sitting on the runbooks. Okay. Yes, I'll go pay for a therapist; rather, I'm glad that my insurance helps cover for a therapist.  ANA: I agree. [laughs] ADRIANA: I want to zoom in on the therapy stuff because I've been in therapy before. I think it was one of the greatest things that I could have done for myself was just seek someone out when I was in distress. And I love that we're able to talk about that freely here because even though I think there's a lot more openness towards talking about mental health in tech, I think we still have ways to go. It would be nice to have...and I think employers are definitely starting to provide support around mental health. But I think there's always more that can be done.  Back in the day, taking stress leave was viewed as a bad thing, and now people...I don't think we even call it that anymore. We call them mental health days. And employers are starting to embrace that with the realization that, hey, sometimes we go through shitty times, and we just need some time to reboot the system so that we can be at our best at work and to have that kind of support so that we can continue to perform and not end up at the hospital because we're stressed out. I think that's really, really important.  But I do wish that there was a little bit more that can be provided because I think a lot of employers will cover up to a certain amount of money in terms of therapy bills, but it's not 100% coverage. And we all know that long-term therapy can last years and years and years. That's just the reality of it. And it's good to have long-term therapy, but it's expensive. JEWEL: Yeah. When I was planning out FSA for how much money do I want to have to put into healthcare, a lot of that is just going into paying out-of-pocket expenses for therapy. But like you said, with the employers putting into it, it's like, yeah, it's like paying for the therapy too, like, putting into your healthcare benefits. But also, that time off, those holidays where if you look at other industries, as a sailor, you would get shore leave where your shore leave is going to be a long weekend or whole week off. It's like, oh, you need five days off to make a substantial holiday of it to feel a reset.  ADRIANA: Yes. JEWEL: So it's like, yeah, take off. Within the tech industry, we do have this very Christmas to New Year's-centric holiday, but it's like, oh, the slowdown that happens in December, we can all jointly agree let's take a day off, or like, let's take a whole week and not just a day. Where it's like a day off is good for a day trip or get some more groceries, whatever. It's like, no, you want to take a little substantial time.  But then you also need to have that psychological safety of, like, my manager will help me with redistributing my labor, or I'm not going to come back to a fire. I will be able to trust in my co-workers to take care of stuff that comes up. Or just be like, you know what? We're just going to reschedule stuff to later. We're just going to cancel some stuff that does not feel more important because you need to take that rest period to be able to continue doing work.  It's like, yeah, this is the capitalist pitch of if you want to get the most out of your staff, you don't run them into the ground. You give people some rest time. You let people sit down and chit-chat and take a couple of days off and not think about work a decent amount of the time. ANA: And it takes, I think, having a lot more of those conversations like making sure that it just becomes normal in many organizations, in many teams of how folks are doing. What I always say is that it kind of comes from the top the same way that we see reliability that unless leadership is the one pushing it down, like, we need to build stable, reliable systems, it's not going to happen. If we're not saying we need to build a team that actually cares about humans, it's not going to happen. And you need to take into account workload, on-call rotations, how many pages they're getting. Like, how much are they getting pulled into incident post-mortems? And I think not a lot of that happens.  And I think one of the things that I'm starting to see again, or maybe it's just more common now, that when you do end up in crisis, is that you're broken, that you've broken yourself. And I myself also had that belief that when I burned out of tech, I was broken. And it's just like; it's just your system telling you to slow down and that you need to do things differently. But it's that shift in mindset that's going to get us to make a change in the industry. JEWEL: Yeah, that's very much previous pain informing improvements of behavior where I'll have these long-running incidents, or I'll have incidents that'll go until, like, 7:00 or 8:00 p.m. And we still have to get up in the morning and deal with it. And sometimes I'll go, I'm prepping a large rollout, and let me think back to, oh yeah, do I have the energy to stay up all night? Nope, nope, I do not have that energy. I have suffered already before.  With that memory of that pain, that informs me to go and say, "Hey, SREs, how can I avoid getting paged for this system in the middle of the night?" Or, like, if there is something that breaks, is it p-zero severity? Or is it the highest severity? Is it mid-severity? Can we set up our paging with respect to that of, like, oh, here's the lowest severity escalation schedule that doesn't fire outside of business hours? Great. Let's use that instead of routing everything through the respond immediately to your pager. ANA: [laughs] ADRIANA: Totally.  JEWEL: And that's one thing I've been glad to see with this or getting more and more mature is defining those SLOs and SLAs that reflect severity levels of, like, is this impacting 100 people, or 1,000, or 20,000 people? React appropriately. ANA: I think people forget how much of an impact fine-tuning alerts can actually be on the mental health of a team and that this should actually be due quite often the same way that folks are revisiting service-level objectives. It's just like; it's a constant need to be evaluated.  ADRIANA: Yeah, it's a maintenance thing. I mean, in some ways, it's like going to therapy. Sometimes you go because there's something that triggers you needing to go to therapy. But then there's other stuff where you continue to go to therapy for continuing to maintain your mental state, and I think SLOs can be looked at in the same way. And when we spoke with Alex Hidalgo about SLOs, one of the big takeaways was your SLOs aren't set in stone. And as your application evolves, they're going to continue to evolve, and that's all right. That's what you should be doing. JEWEL: And that's also where I appreciate my management for enforcing the social side of that of, like, oh, you need to host these therapy sessions. And where do we do it at work? We have our every-other-week sprint retro. We have our post-mortem sessions for big events, and then potentially scheduling more where it's like, oh, we see a recurring trend; let's do a roundup. Or we have our operational excellence meetings every couple of weeks or every month where we go through and talk about, like, what's been happening recently? And how do we feel about it? Can we do better? Where it's like, this is group therapy in a technical setting for our systems and ourselves. ADRIANA: Yeah, totally. That's awesome. One final point that I want to make along the lines of making sure that you have support from...just going back to the previous point of making sure you have support from your organization around mental health, I feel like, going back to what you were saying, going from IC to manager I feel that when you get into a position where you can have that kind of influence on the well-being of your team, as a manager, drawing on that IC experience of like, hey, I've been in situations where I haven't been supported in my mental health, and as a manager, I can do that.  Or I've been in situations where I have been supported in my mental health, and I've seen great results from that, and as a manager, I can also do that for my team. The more people coming into positions of management having that, the better. I just want to go back to Jewel's comment about supportive managers with respect to mental health and supportive organizations with respect to mental health.  And one of the things that I think is super important is that as people in their careers come up as managers, I feel like this is a really great opportunity for them to effect really positive change on an organization where you can draw from personal experiences whereby, like, hey, I was on a team where my mental health wasn't taken into consideration, and therefore I don't want that to happen. I want to do better for my team. Or the converse, where you've been on a team where your mental health was taken into consideration. You had a kick-ass manager, and now you have some behavior that you can emulate.  But basically, I want this to serve as a call to action for anyone who's coming up as a manager, whether it's for the first time or you're going back and forth between IC and manager. I feel like this is a really good opportunity to really make some positive changes in how an organization treats mental health. The more folks in management continue with this pattern of positive behavior; I think the more that we can start seeing more positive changes in the industry around mental health. JEWEL: Yeah, that's really nicely said. And yeah, definitely, as more people get into the industry and get into supporting your mental health and supporting your team's mental health, it's like, yeah, when you're the manager, you're the one who gets to make those decisions of when are we going to be hosting these daily and weekly meetings? You seem to be working too hard. You need to take a break. Or figuring how to finesse that...saying it softly where some people will be like, "I'm fine," and be like, "But are you though? ADRIANA: [laughs] JEWEL: Would you like to review your velocity? And you can come to the conclusion that you actually need to take a break." ADRIANA: Yes. ANA: Definitely on tracking those PTO days of, like, "You haven't taken days off or go get an Airbnb somewhere and work somewhere else remotely."  ADRIANA: Yes. ANA: Like if your job permits it, those things like it. Or I would even add, like, maybe even just posting as a resource within the engineering organization, like burnout index. Like letting folks understand if they're possibly near burnout and, what they can do about it, what burnout actually looks like. Having that conversation that if you're burnt out for too long, you might fall into a depressive episode and such. We have to be a lot more proactive about these conversations, the same way that we've been preaching about reliability. Like, what work can we be doing now to not be stuck in an ouchie or worse later? JEWEL: Yeah, and it's really fun to talk more and more about that because, like...we haven't talked much about it today, but I got diagnosed with ADHD several years ago. And that's mostly because I saw people like Rich Burroughs talking in public on Twitter about ADHD coaching and getting diagnosed and seeing other people in my tech sphere talk about ADHD and going, oh, what they're talking about, that's me. I'm reflecting those same behaviors.  And this is a particular approach to working well with, like, this is how my brain works. Here's how I can work with my brain. So it's like, it's no one size for any service. It's no one size for any person of how to make things work well. So just talking about it and putting it out there where it's like, oh, just, you know, mention this is how this impacts me, and other people just realize, oh, that is me. ANA: [laughs] JEWEL: And then figure out, like, okay, how do I work better? Like, for me, it's I put a lot of effort into...I now take meds. I now do better practices. Like, at the start of my day, I pick my top three things to work on. I tell my co-workers like, hey, if you want this thing to get done, maybe schedule a meeting where we just sit together and work through it together because otherwise, it might not get done. So you and I would both appreciate if we just sat down together and did this because that's what works for me. And I see this, especially within SRE, where I feel like a lot of SRE culture is actually codifying ADHD coping tactics. ADRIANA: [laughs] ANA: Whoa, I love that. [laughs] JEWEL: Yeah, because it's like, oh, we don't want to do the same work three times. Let's make the computer do it.  ADRIANA: Yes. JEWEL: Or like, oh, there's something broken, and we're good at fixing it. Yeah, let's go fix it. But also having that cognizance that you might not recognize that you've been working for too long because you've really focused on this thing and you need to cycle out, like, somebody else will just come from a rotation, and you have to go hand off. And it's like, oh, yeah, okay. Okay, thanks. Thank you for having some external guardrails on my own brain. ANA: There's definitely a lot that SRE guidelines, and rules, and goals actually teach you, like, if you are to embody those into your life in a way to live a fulfilling life. I've said in multiple [laughs] podcasts those are the type of things that keep me up at night. How is it that you find that parallel of, like, am I doing my hobbies, my habits? Am I running towards my goals and making sure that I stay focused through the day while doing so? Which is kind of like what our complex systems that we're on-call for have to do. [laughs] ADRIANA: Totally. JEWEL: It is fun to be a complex system. My partner will sometimes go like, "Remember, you are a complex plant. Go get some sunshine and some water." ADRIANA: [laughs] JEWEL: It's like [crosstalk 44:34] ADRIANA: Yes. JEWEL: Yeah, okay. ADRIANA: It's good to be reminded of that.  ANA: I love that. You are a complex plant.  ADRIANA: That's going on our Twitter, and Mastodon, and all the things. [laughter] JEWEL: Pinned post: you are a complex plant. Get some electrolytes. That's what plants crave. ANA: Yes. ADRIANA: You need sunshine and water. [laughs] Love it. ANA: So with that, we would like to thank you, Jewel, so much for joining us in today's podcast. We loved talking all things from SRE to on-call, ADHD, taking care of our mental health.  Don't forget to subscribe and give us a shout-out via all the social media such as Mastodon, Twitter, LinkedIn, Instagram via oncallmemaybe. And be sure to check out the show notes on oncallmemaybe.com for additional resources and to connect with us and our guests on social media. For On-Call Me Maybe, we're your hosts Ana Margarita Medina...  ADRIANA: And Adriana Villela. Signing off with... JEWEL: This is Jewel wishing you all peace and love for the humans we collaborate with and code to support that peace. Thank you so much for inviting me today. ANA: Whoo. ADRIANA: Thanks so much for coming on. This was great.

About the guest: Jenny Gee-Link is a Quality Assurance Automation Engineer at Tangerine. She has an extensive background in Mobile Applications, Wireless Technologies, and Telecommunications, having previously worked at large Telecommunications companies such as Motorola. When she’s not working, she always seems to have something on the go. Her spare time is spent on volunteer work with her kids’ school and on the board of directors for the local daycare, picking up another language, tinkering on the piano, and even out biking on the local trails. Binge-watching the latest shows? Nah. Life’s way too short for all that! Find our guest on: LinkedIn Find us on: On-Call Me Maybe Podcast Twitter On-Call Me Maybe Podcast LinkedIn Page On-Call Me Maybe Podcast Mastodon On-Call Me Maybe Podcast Instagram On-Call Me Maybe TikTok On Call Me Maybe Podcast YouTube Channel Adriana’s Twitter Adriana’s Mastodon Adriana’s LinkedIn Adriana’s Instagram Ana’s Twitter Ana's Mastodon Ana’s LinkedIn Ana's Instagram Show Links: Tangerine Bank Perl Motorola University of Toronto Motorola StarTAC Phone Motorola Razr Phone Blackberry T-Mobile Sidekick MSC (Mobile Switching Center) Base Station Additional Links: Gen Z embraces flip phones Transcript: ANA: Hey, y'all. Welcome to On-Call Me Maybe, the podcast about DevOps, SRE, observability principles, on-call, and everything in between. I am your host Ana Margarita Medina with my awesome co-host... ADRIANA: Adriana Villela.  ANA: Today, we're talking to Jenn Gee-Link, who works at Tangerine as a Quality Assurance Automation Engineer. We are very excited to have you here today. One of the first questions that we have for you today as you join us is, what is going to be your drink of choice? JENNY: Thank you. Thank you for inviting me. Thank you for having me at your podcast. I'm honored. I'm happy to be here. Regarding your question as to what is my drink of choice right this moment, just to get over my cold, my pseudo cold, my sore throat right now, it's hot water, hot tea, hot cocoa if it's not too hot, too, too hot and burning my tongue. But that's pretty much my choice for now until I get better. [laughs] ANA: I've been drinking a lot of tea. My roommate was sick. My partner was also coughing. And I'm just like; I'm just going to drink tea with honey, like, protect myself. ADRIANA: Yes. ANA: Stay hydrated and, like, just [laughs] [crosstalk 01:28] ADRIANA: Protect the vocal cords.  ANA: Yeah.  ADRIANA: Totally. ANA: What about you, Adriana? What are you drinking today? ADRIANA: I've got fizzy water. It's Bubly sparkling water, and it tastes like cherry, so... JENNY: Love that.  ADRIANA: It's an improvement over my usual water drink I have whenever we record, but I do have water on standby as well, so... [laughter] ANA: I'm definitely like the non-fun one for the day. I have water, just regular water for the day. I was going to make a mocktail, and I was like, maybe for the next recording. [laughs] ADRIANA: That would have been fun. [laughs] ANA: We shall see. So, Jenn, we've had guests from all walks of life come on to our podcast and talk about their journey. And we love hearing how people got started in tech. Like, what caught your eye? JENNY: That's actually a very good question because how did I get here? I honestly don't know how I even got here. [laughter] ADRIANA: My favorite answer to date, yeah. ANA: That's okay. JENNY: I don't know how I got here. Oh, I remember there was one time, I do remember this; way back in school, I was applying for just a summer job in the U.S. and just for the fun of it. I was applying back in December for a summer job. Why? You're probably like, why? Why do you apply for summer jobs in December?  Well, apparently, the Americans they do all their summer job hiring in January for preparation for summer jobs starting in May. So they do all the interviews in January and everything. They start prepping all HR people to pull you in wherever you are in the world so that by March 1st, they already have people lined up ready to go. So they fly them in ready for April 1st, May 1st. And then I'm there, voila, for a summer job.  Anyway, I didn't know, and I was going like, okay, here I am in computer engineering. I didn't know anything. I knew nothing. And I saw in this job posting, "Do you know Python?" I go like, hmm, I don't know Python or anything like that. Oh, oh, oh, and do you know Perl? No, I don't know Perl. [laughter] Okay, let's look it up. Let's Google. Let's Google it up. Then I was like, okay, bam, on my resume, I can say I know Perl. [laughter] I can simply say on my resume, "I know Perl. I know how to do that." [laughter]  And lo and behold, I got selected for an interview. And they say, "Hey, you know Perl? You must be an expert on this." I go like, "Yeah, sure. [laughter] Give me a job, please. Please do. Please. Please do." They go like, "Great. You're coming in May 1st you start. ANA: Oh my God. [laughs] JENNY: And I go like, [gasps] yeah, that was it. That was absolutely it. They flew me in. I'm like, "Okay, I know Perl." "Okay, but what year are you in?" "Third year." "But how much Perl experience do you have?" "I read this document. [laughter] That's all I know." And they said, "Well, good enough. You're here already. It doesn't matter. [laughter] Oh, by the way, you get paid $22 an hour." "Oh my God, this is awesome." [laughter] ADRIANA: You're rich, right? By university standards. [laughter]  ANA: As a college student, I'm like, well, I mean, as a college student, but just in general, that is really good money for a first job when you're anywhere. [laughs]  ADRIANA: Yeah. Totally, totally. JENNY: Yeah, absolutely. That's all I did. On the job posting, they were looking for someone that knew Perl. I really spent 40 minutes reading that document [laughter], and then I can say on my resume I know Perl, which is true. It is true. Did I know how to run it? Yes, Perl blah, blah, blah.pl. Okay, what does it do? Hmm, [laughter] I don't know. And I got paid 22 bucks an hour.  And I said, "Oh my God, this is awesome." I got flown in. I had an apartment. Yeah, my apartment was covered by my income. That's okay and everything. And I had extra money to spare after being paid 20 for 40 hours a week, sure. And I had more money left over at the end of the summer. And I go like; this is awesome. So that's how I started off with. [laughter] ANA: You mentioned you were studying computer science, computer engineering. JENNY: Computer engineering, that's right. That's right.  ANA: So, what got you into computer engineering?  JENNY: Oh, because I didn't get into other things. [laughter] ANA: This story just gets better. [laughter] ADRIANA: This is like the best story. [laughter] What did you apply for?  JENNY: Civil. [laughs] ADRIANA: You applied for civil engineering, and they put you in computer engineering?  JENNY: [laughs] ADRIANA: See, this is the state, like, Jenn and I went to school, like, we were in university together. We were in different programs. I remember around that time, computer engineering was the hot stream of engineering that everyone wanted to be in. And it was funny because you ask half the people, "Why are you here?" "It pays well." "My mom wanted me to study computer engineering," [laughs] or I guess in your case, "I got rejected from civil engineering." [laughter] JENNY: My physics mark was really shitty, and they were like, okay, you're not going to get in here. [laughter] And then it was like, but why didn't you apply to chemical? Because chemical I was actually really good at, I was getting top marks. And chemical was my third choice after computer. [laughter] So civil didn't want me. [laughter] ANA: Almost. You almost made it to chemical. ADRIANA: That is so funny. [laughter] JENNY: Civil didn't want me, so computer took me. And then chemical, I guess they would have wanted me had they gotten the opportunity, but hmm -- [laughs] ADRIANA: Computer engineering got to you first. [laughter] JENNY: It's sad. It's really sad. [laughs] So that's where I ended up with. But why did I stick with it? Because after school, the company that I eventually got hired into, Motorola, I was put into a team for not just the cellular networks but the whole telecom network, the whole entire framework. And that was an eye opener for me as to how your cell phone calls started off with on that little handset going all through the base stations to the MSCs and then to the mainframe where I actually set the call setups and everything. So that's ultimately where my interest was coming from, just from my first job.  In school, you learn. You kind of are taught it and everything, but you really don't know how it works until you're actually getting your hands deep into it so that you know that, hey, this call starts at this stage, to this stage, to this stage and eventually gets to this. And then wherever I am in the world, eventually, it gets routed to wherever, here to there. And everything comes right back out to whichever...whether you're talking to someone on the telephone line, or to someone else's cell phone, to an IP phone, or anything like that.  So that kind of communication, I thought, was really, really cool, and that's where I started off with. And I'm kind of...even though I'm now in banking, and I'm not quite touching that right now, I'm still very well-versed with how things are working here and there and how things are working in the industry. So that's where I'm at. [laughs] ADRIANA: That's so cool. And the other interesting thing, too, from when you started in telecom, is that it was probably around the time when cell phones were finally starting to become popular where it wasn't just the "use cell phone for emergency calls." It was like, oh, I can use it to talk with my friends. And then texting was starting to become popular on those little, [laughs] tiny keypads where you had to press the key like three times to get to the letter C. [laughs] JENNY: Right. Right. ANA: I miss those days. ADRIANA: [laughs] Yeah, the simpler times JENNY: I worked on the original StarTAC phones, I did. ADRIANA: That's so cool. [laughs] JENNY: I worked on the original Razr, the Motorola Razrs. I worked on the original --  ADRIANA: Oooh, that was a sleek phone. JENNY: Oh, it was a sleek phone. ADRIANA: Super sleek. ANA: I owned the Razr. ADRIANA: [laughs] ANA: I also owned the other Razr that was like a non-flip phone, the pink one. Yes, that was...I remember being like; I must own this phone. ADRIANA: Oh, pretty. [laughs] ANA: And it was one of the first phones that had integrations with Apple, so it was like an iPod phone. And it was like the coolest commercials ever. And I was like, oh my God, yes. [laughs] So I'm a Razr girl. JENNY: I'm glad to hear that because it is, like, if you look it up on Wikipedia, it was, or is, was, the most bought phone in the world, and it's still now; why? There are different reasons, but it was the most popular phone. I'm not saying the iPhone is not. It is very popular now. But if you look it up, you can see it's right up there. ANA: Have you all heard that there's this group of Gen Zs that are trying to make it a thing to bring flip phones back?  JENNY: Yes.  ADRIANA: Yeah, it's hilarious.   ANA: [laughs] How do you all feel about that?  ADRIANA: I mean, I loved having my flip phone. I loved having a flip phone. It was so awesome. And I was really sad, like, I love my iPhone, but I was really sad that it was no longer a flip phone. There's something very satisfying about just flipping it open to make a call. [laughs] ANA: And on that note, I personally miss the sidekick. Being a millennial, the sidekick was like, full keyboard. ADRIANA: [laughs] ANA: Moving on from Blackberry, I was like, oh, look, it's got that cute, like what? Like 280 flip of a screen. [laughs] JENNY: Oh, that's right. ADRIANA: Oh, yeah, yeah. I never used one, but I remember there was a period in what? Like the early aughts where it was popular. ANA: Like 2005? ADRIANA: Yeah, 2005 to 2000 and, I don't know, 10, 8, something. ANA: 23? [laughter] JENNY: So, ultimately, yeah, phones I actually worked on. I still work on them now. But even back then, I even knew about the whole underlying infrastructure as to how all the calls are going through, how to intercept, to change to look for certain things. So every single keypad that you make, I see it. If you're in the back end, you could see every single thing. Oh, someone's putting in their credit card information into their phone; I see it. You can see it. It's all there.  And it's really interesting when people sort of say, "Oh no, I'm keying in my credit card. I'm keying in my birthday and everything." Well, don't worry, well, for the most part. [laughter] It's pretty safe. It's pretty safe. Unless you have some judge that has put out a request for this sniff, we need to get this information out. So I've gone around to many different parts of telecom, at least specifically for cell phones and such. It's interesting how it all goes around the whole circle, how things get around, and people see how it works and everything.  ANA: That's super cool. ADRIANA: That's awesome. So then, how do you go from cell phones to testing software? JENNY: That's really interesting because one of the biggest changes that happened to me about 10...yeah, ten years ago is the need, at an infrastructure level, as how can we emulate how many calls are happening at the same time? I mean, right now, our biggest problems, at least for the telecom level, is to say, how do we reproduce a high call load? How do we reproduce that? And how do we reproduce not just at one spot, how do we reproduce it at a state level, at a tri-state level?  And that need was mandated by the government to actually prove that, hey, does the software work? Can it handle that load? Whoever's buying the software they're paying billions of dollars for this, but they need proof to say that can it handle it? Will it die at the utmost need of time? I mean, we wouldn't want something like that to happen. We've already seen instances where networks have gone down, and I'm not going to name them or anything like that or name instances, but we've already seen it. We've experienced it, and it's pretty crappy. It's really shitty when hey, the cellphone network or telecommunication network is down because there's an overload.  So that's one of the reasons why that came out, why I came out in testing now, because there was an absolute need where the buyer wants proof that can it handle it? We need to show it. The developer has claimed that it works, but they haven't tested it. They haven't tried it. They really haven't. I mean, one cell phone call, sure, that's one. ANA: [laughs] JENNY: But you need to make a couple million. And it's not just for one at one point; it's like a couple million for over a period of 18 hours, 24-48 hours, more than that. And that was the need, that was absolutely the need. And how much testing is done right now, I'm not going to comment. [laughter] But there is definitely a need for testing. Absolutely. ANA: What are some of the practices that go into doing these types of testings? Like, is it all using stuff that's built internally that's very specific to your organization? Or are there bigger projects that are used or methodologies?  ADRIANA: Like frameworks. ANA: I ask in the sense that I come from the world of chaos engineering. We do a lot of making sure that reliability is top of the mind and inject failure wherever possible. And a lot of organizations always pair it with something like load testing. But I haven't been too close to folks that actually do that with telecom. So I'm like, ooh, does it kind of go through a similar type of process? JENNY: In many ways, it does. We do realize that there is no way to really reproduce the exact same scenario that we've seen in telecom where things go down. We can only speculate based on the logs that we are able to retrieve and search and see that, oh, if this happens at this time, this will happen, et cetera, et cetera. We do use a number of in-house tools because we realize that there has been a number of tools that are provided that are open source, but they're very limited because it's only specific for certain scenarios.  So if we need to produce a load, yes, we can produce a load. We just simply send out a whole bunch of basic, I would say...we're not even pinging; we're not doing that. But even just calls, just not constant calls but when we make that call, yeah, a response. Yeah, that's great. But that's not really the scenario that is actually happening out there. So whenever someone makes an actual telephone call, we need to have that one signal going from cell phone to base station, the base station to whichever MSCs that it has to go through and everything. And then we have to go through the whole entire call flow going through there.  So, yeah, we can kind of automate where we inject a call out. But then we also need to communicate with different components to actually have that call setup happening, and that is a lot more difficult. We realized that we can't do the full call setup because, generally, when we test, we're not testing for a million call setups happening at once. Because the only time I can think of that people are doing that is there is an emergency 911, and literally, there's a volcano that comes out that says that [laughs] "Heeey, I'm going to explode." And then there'll be a million people making that phone call who are like, "Call 911." That doesn't necessarily all happen. [laughs] ANA: And now I'm just curious, like, is one of the other high traffic days like New Year's as well for telecom, or had it been in the past and maybe not as much in current times? JENNY: Not necessarily because now, with the introduction of...you know what? The funny thing is that there have been less calls being made now than there have been before because everyone's now Instagramming each other. They're all text messaging. There are actually more text messages that go out than calls, kind of funny. ADRIANA: Yeah, yeah, that's awesome. [laughs] I definitely believe that. I hate talking on the phone. [laughs] Like, text me; do not call me. [laughs] JENNY: Exactly, right? No one leaves voicemails anymore, and no one calls. ADRIANA: No.  JENNY: Yeah, it's all texting or even just messaging someone using whichever messenger you want. I'm finding that more and more teenagers are using Discord to communicate. Even though they're not playing games, they just simply say, A, I already communicate with you when I'm playing games. I'll just message you. Just DM me, and then blah, blah, blah. And I'll send a message going like, "Hey, this is what I'm doing on New Year's Day." So that is happening. Regarding call volumes, if you really want to know, I'm finding [laughs] that for call volumes, it's usually if a radio station is having some kind of giveaway. During those times when they're making a giveaway...at 8:00 p.m., call me at this time; 80th caller will win this. Yeah, that's when the spike goes up, but those are actual calls. If you're talking about calls, that's when it's happening.  ADRIANA: That's hilarious. ANA: Interesting. [laughter] JENNY: It's really funny. It's really funny. ADRIANA: [laughs] So the last remaining relic is the call into the radio station. I guess some things never change. [laughs] JENNY: It's really funny. It's really, really funny. But that's a niche form of communication. That's only for radio stations. The main communication is for this messaging that people are messaging, or when people are actually video and streaming whatever they're doing and showing their friends, and they're broadcasting from there. That's the big one. ANA: So, like any video calling like Zoom, FaceTime, like Discord video. JENNY: Yeah. ADRIANA: Or live streams. ANA: All the Twitch. [laughs] ADRIANA: Yeah, yeah. JENNY: We're not really using our calls to communicate with people. It's been changing. You can think of it as it's now data being passed on. It's no longer calls anymore. So you use your cell phone not to call people but to transfer data to people as you're communicating. And that's the change in people's way of communicating over the past decade or two decades. ADRIANA: That's so trippy. JENNY: It's really funny. It's really interesting. ADRIANA: You don't even realize until you take a minute to sort of reflect on it. And then you're like, oh my God, the way that I was communicating 15 years ago is not even the same way that I'm communicating now. It's super wild. JENNY: Right, right. So now people are using their cell phones to bank. [laughter] So why not? But to even communicate, to talk to people...no, no, not communicate but to even just send videos to people, TikTok, there we go. That's just interesting. That's very interesting overall. ANA: [laughs] ADRIANA: I wanted to switch gears a sec and talk about the fact that...I don't think this has ever come up in our podcast yet; like, you're a working mom in tech. But we've never...I don't think we've ever talked to anybody or at least talked about being a working mom in tech. And I wanted to get your thoughts on how that has been for you. [laughs] JENNY: It has been hard. I'm well aware of the sacrifices that I've had to make over the years, and because they'll...right now, even the kids know that mommy's on a call right now. I need to stay quiet, and they really don't know why. But they know that mommy is busy working on certain things to make it better. Now it's gotten better for me. But back then, I really had to make severe sacrifices back then which I do regret. I kind of wish I didn't have to do that.  And it's been very, very tough. Why did I make those kinds of sacrifices is more the fact that I've felt that I needed to prove my value, my worth. And I don't want to be that junior engineer forever middle person. I need to be the senior after so many years. And I need to prove my worth by getting to that level that I know something more than dipshit over there. [laughter]  ADRIANA: I feel ya. I feel ya. JENNY: It's been tough. And that's also another reason why I've taken the job at the bank because I needed to slow down so that I can actually do the things I should have done back then. You're probably thinking, well, what did you do back then that was so fast, that was so necessary? I don't know. I'm not going to continue on with that part. ANA: [laughs] It's okay. ADRIANA: I would like to comment that I remember when I came back from that leave with my daughter and put her in daycare, and I swear to God, [laughs] it feels like every week they're sick. And I can't tell you the number of days, the number of times I had to leave work early. One time I just dropped her off at daycare. I stepped into the office, got a phone call, "Hey, she's got a fever. You got to pick her up from daycare." And it's like, hi, guys. Bye, guys. [laughs]  And feeling like, okay, I got to take care of my kid but, holy shit, I feel so guilty that I'm having to walk out of the office right now. People are judging me. I felt like people were judging me. I don't know if they were, but I certainly felt like it. They probably think I'm not pulling my weight. So I felt guilty. I felt like I wasn't doing a good enough job as a mom because I was thinking about work when I was with my daughter. And then I'm thinking about my daughter when I was at work.  I feel like you can never win compounded with depending on where you're working; if you don't have a welcoming environment or nurturing environment that makes you feel like it's okay to do that or it's okay to be human, I think it makes it way, way worse. So yeah, that was my personal struggle. JENNY: Absolutely. Absolutely. I was working with a team all-male team. My manager was also a male. He used to work for the military. So he operated the whole team as the military.  ADRIANA: Oh geez. JENNY: So it was really hard. It was hard. It was heartbreaking when that came about. And I was working really late hours. Kids never saw me until 1:00 a.m., 2:00 a.m.  ADRIANA: Oh, shoot.  JENNY: My husband was the one who had to leave to pick up the kid. And then he said, "Okay, tomorrow, you're working from home." And we sat there, and we were like, "Okay, it is what it is." And it was very difficult. I mean, you had the one, I had the three.  ADRIANA: Yeah, yeah. Yeah. [laughs] JENNY: And it was very difficult. During that time, I swear I went crazy. I went crazy during that time. Out of desperation, I hired a cleaner to my house because everyone was getting sick. And when I hired her, I didn't bother asking for references. I didn't ask for...beyond her name, her phone number, or anything like that; I just said, "How much? Okay." And I handed her the keys to my house. I said, "I'll see you whichever day. Please come clean."  And she said, "Oh, am I cooking dinner?" I said, "No. Your sole job is to come every week and wipe down the counters, bathrooms, kitchen countertops, so that...oh, and then also wash all the sheets in the bedrooms. Just wash them all so that no one can spread any more germs." And that was her sole job. She was being paid 70 bucks a week just to come to my house to clean, and that was it. I felt shitty because I didn't know her name. I didn't remember her name. My husband's going like, "You did what? Gave keys to the house?" ANA: And I think in immigrant homes, it's even harder to understand outsourcing stuff that our families would do or, like, I tell my mom certain stuff that I spend money on. She's like, "That's so expensive," or like, "I could be doing that for you." And it's like; literally, corporate America is so different. JENNY: Yeah, absolutely. The thing was that it was time, my time, my sanity. Even when I had the kids in daycare, I was still working late hours, keeping up and everything. Was I actually physically in the office? Yes, I was because I need to show that I can actually pull my weight. And I did. And every so often, my husband would be messaging me, going like, "Are you okay? It's 1:00 a.m. Are you okay?" And I said, "Yeah, I'll be leaving in about two hours, three hours. I don't know; I'll be there." Really crazy time.  And after about four years of doing that, I noticed that the kids were also feeling the effects. Their learning was delayed, the reading was delayed. And I felt like a shitty mom. Of all things, you don't know how to read. You don't know how to read, all right. I was desperate for anything, any type of book, some magical book to have them read basic, "It is a man. It is an apple. Come, go, come, go, come," like those basic things. And it was absolutely a struggle, absolutely a struggle. ANA: I'm sorry.  JENNY: There's nothing to be sorry about. I realized that over time that these sacrifices had to stop, and it did happen right before COVID. And that's why I walked away. That's why I'm at a bank. [laughter] There's nothing wrong...I'm not saying that I went to the bank because it's an easy job; it's not. But it's realizing that I cannot go down that route again. ADRIANA: Like, you're in a place where there's better work-life balance so that you can maintain sanity, take care of yourself, the self-love, self-care that we all deserve. JENNY: Absolutely. So if all of a sudden the bank starts noticing that I disappeared for two hours in the afternoon, they will ask, but I say, "No, I needed to go for a walk. Yeah, I'm done. I needed to go for a walk. I really have to...I'll make up for the hours later on tonight. But for those two hours [vocalization]."  ANA: [laughs] You need a break. JENNY: I really have to. The other place I could walk away, and I did do that on a regular basis. But then I'll be back in half an hour, and then I'm right at it going at full blast. ADRIANA: And you're probably panicking while you walk away; oh my God, I've got all this shit to do. [vocalization] [laughs] ANA: Yeah, they're going to say something.  ADRIANA: Yeah, the whole butts-in-seats thing. [laughs]  JENNY: I went crazy. I did go crazy. I scared off a number of Jehovah's Witnesses. [laughter] It's actually quite scary. It's very scary. I imagine normally, people run away from Jehovah's witnesses, and they're usually happy to talk to you. But no, I swear, I scared off a group of Jehovah's Witnesses when they actually showed up at my house. I scared them. I really did scare them. [laughter] ANA: Rough times. What do you think the industry should be doing to support working parents more, like, if you could give advice to folks that are in leadership positions to build cultures that people feel like family can come first, that you are a human, and this is a job, but you're able to attend to your other needs? JENNY: I'm not sure. Maybe it's because I was really pushing myself to keep up with the other senior developers in ensuring that the whole entire development pipeline keeps going. And I'm not talking about actual pipelines but the whole entire process it all gets through. I find that if management actually says, "It's okay. If you need to leave, it's okay. Someone will cover you. Don't worry."  But then if you're saying that, "Oh, but really, I can do it myself," and then they just go like, "No, no, no, not really. You take care of yourself. We don't want you to the point where you're going to really throw yourself off the cliff because you've gone...you're so worried that this is not working. This other thing is not working. Please take the time. Take care of yourself first, then your family. Work will be always here tomorrow. Don't worry." And I think that is the most important thing that they actually can say that. And I don't see that very often with management. They'll say, "Oh, go home. You take care of your family, yeah, yeah, yeah." They won't say that "Work will be here waiting for you. Don't worry. It won't go away. It won't explode. It won't disappear." I think that's more of the thing. "It's okay. It's not a big deal. We'll handle it. We'll push out the schedule. It's no big deal. It'll be here." ANA: Actually take actions to support their employees versus just words.  ADRIANA: Yeah, we definitely need more of that. Absolutely.  JENNY: Right. And then later on, not judge you based on the fact that, oh, your bonus this time won't be that big because you missed your deadline. Everything got slipped. But I think that's the expectation; it's supposed to slip. That deadline you just made up is just made up. It doesn't matter. You pulled it out of your butt, out of thin air, last week. Who cares? It's just a date. What is it? It's just a number; it's just that. And that's the problem is that they should realize it's just a date. It's okay. ADRIANA: Yeah, I totally agree with you. And that's definitely been a personal gripe I've had around estimation. It's like, oh, you must stick with what you estimate. It's like, dude, I estimated. It's a best guess. If I were accurate, I wouldn't be working here. I'd be a millionaire or a billionaire. I wouldn't be sitting here talking to you. [laughter] JENNY: Yeah, absolutely. Yeah, those dates are just dates expected to be slipping. ADRIANA: Yep. Shit happens, right? [laughs]  JENNY: Yeah, shit happens. ADRIANA: You can't account for everything. [laughs] JENNY: You can't throw bodies at it and expect to actually keep that date. It's just not possible. ANA: I think a lot of it, too, is like leadership leading by example of finding workplaces that have leaders who are also parents, underrepresented folks that are able to create a culture for their employees that they feel like they can bring their whole selves. Or they're actually taking spring break off because they're spending time with their kids at home or actually just signing off to do dinner with them and things like that. Those little motions really do go a long way. ADRIANA: Yeah, exactly, exactly. Well, I think we're coming up on time. So I want to thank Jenn for joining us today. It has been super awesome chatting with you and getting your perspectives. And honestly, it's so refreshing to have a candid conversation about these things because we don't have enough of those conversations. Certainly, I'd say in the corporate world, that's the type of conversation that we don't get enough of, and you kind of take for granted in the startup world.  So I think it's nice to surface these types of conversations because these are problems that still occur, and they need to be addressed. Like, we're not robots; we're humans. We all need TLC. We need to all take care of each other. So thank you for sharing your thoughts and your story with us and with our audience.  Don't forget to subscribe and give us a shout-out on social media via oncallmemaybe. We actually just joined Mastodon, so you can find us on Twitter and Mastodon on oncallmemaybe. Be sure to check out show notes on oncallmemaybe.com for additional resources and to connect with us and with our guests on social media. For On-Call Me Maybe, we are your hosts Adriana Villela and... ANA: Ana Margarita Medina. Signing off with... JENNY: Peace, love, and code.  ADRIANA: Woo-hoo! ANA: Woo-hoo!

About the guest: Daniel Kim (He/Him) is a Principal Developer Relations Engineer at New Relic and the founder of Bit Project, a 501(c)(3) nonprofit dedicated to making technology accessible to underserved communities. He wants to inspire generations of students in tech to be the best they can be through inclusive, accessible developer education. He is passionate about diversity and inclusion in tech, good food, and dad jokes. Find our guest on: Daniel’s Twitter Daniel’s Linkedin Daniel’s Twitch Find us on: On-Call Me Maybe Podcast Twitter On-Call Me Maybe Podcast LinkedIn Page On-Call Me Maybe Podcast Mastodon On-Call Me Maybe Podcast Instagram On-Call Me Maybe TikTok On Call Me Maybe Podcast YouTube Channel Adriana’s Twitter Adriana’s Mastodon Adriana’s LinkedIn Adriana’s Instagram Ana’s Twitter Ana's Mastodon Ana’s LinkedIn Ana's Instagram Show Links: New Relic OpenTelemetry CNCF Cloud Native Interactive Landscape Observability Day - @ KubeCon EU 2023 Kafka Cassandra Distributed Tracing for Kafka with OpenTelemetry with Daniel Kim at Kafka Summit London 2022 Tracing Kafka with OpenTelemetry Key Metrics To Uncover the Root Cause of Kafka Performance Anomalies with Daniel Kim and Antón Rodríguez Bit Project Additional Links: Genki Forest - Lychee Sparkling Water Major League Hacking - SRE Fellowship National Academy Foundation Transcript: ADRIANA: Hey, y'all. Welcome to On-Call Me Maybe, the podcast about DevOps, SRE, observability principles, on-call, and everything in between. I am your host, Adriana Villela, with my awesome co-host...  ANA: Ana Margarita Medina. ADRIANA: And today, we are talking to Daniel Kim, who works at New Relic as a Developer Advocate. Welcome, Daniel. Super excited to have you on the show. DANIEL: I am so excited to be here. I feel like I'm on a celebrity podcast, so I'm very excited. ADRIANA: Ooooh, dang. [laughter] I actually remember when you and I met at KubeCon. You were like, "[gasps] Do you host On-Call Me Maybe?" And I'm like, "Yeah." And then you're like, "[gasps] I listen to that podcast." And I'm like, "Oh my God, this is like the best moment of my life to meet someone who actually likes our podcast." So I was super stoked. DANIEL: Also, the title is the best.  ADRIANA: Yes. ANA: [laughs] DANIEL: I mean, On-Call Me Maybe. Shout out to Austin. ADRIANA: Yeah, for real. He came up with a great title. First, we always ask our guests, what are you drinking today? DANIEL: So I'll open the can live on the podcast.  ADRIANA: Whoo. ANA: Whoa. [laughter] DANIEL: So I just opened a can of my new favorite obsession/drink. It's called Genki Forest. And it's a Lychee sparkling water, and it is so good. I've been drinking like four cans of it every day and --  ADRIANA: Oh my God, sounds awesome.  DANIEL: I have to start to ration my stockpile. So that's how good it is. So I could not recommend it more. And it's so refreshing. ADRIANA: I need to find this stuff in Canada. ANA: That sounds delicious. And I think I'm going to have to buy some for myself to be able to share back on socials and be like, "You know what? Daniel was right when he was on our podcast. This drink is amazing," because I'm also a huge lychee fan. ADRIANA: Yes. Yeah, you had me at lychee, honestly, yum. What are you drinking, Ana? ANA: I have nothing fun for today. Today is just water for me. But I am eating raspberry jello. Is that a drink? [laughs] ADRIANA: It's drink-like.  DANIEL: [laughs] ADRIANA: I mean, jello takes on the form of its container. Does that make it liquid? I guess, [laughs] until it solidifies. I've got some fizzy water. I'm just finishing up a can. It's called Bubly. It's sparkling cherry water. And then I've got water on standby because I'm almost done with my can. ANA: Yeah, it's always good to stay hydrated. I'm now kind of jealous of not having a Lychee sparkling water with me. That's all. [laughs] ADRIANA: I'm like dying here because it sounds delicious, and it's given me all the bubble tea vibes.  DANIEL: Yes. I had such good bubble tea last weekend. I love bubble tea. It's kind of an obsession for me. ADRIANA: Oh my God, me too. [laughs] I even make it at home. [laughs] The bill was getting too high with the bubble tea, so I'm like, I need to figure out how to do this at home. [laughs] ANA: Reasonable. I've yet to actually have bubble tea in a while.  ADRIANA: [gasps] ANA: I passed by Boba Guys yesterday in San Francisco, and it's like one of the spots. And I was like, I want some, but I was going in for a dental procedure.  ADRIANA: Ooh, boo. ANA: And I was like, I'm pretty sure I can't have this now, and I can't have it later.  ADRIANA: [laughs] ANA: So I've been thinking about bubble tea since then. So now that y'all mentioned it, I'm like, hmm, good to remember I can't have that yet. DANIEL: [laughs] ADRIANA: When I'm in San Francisco next year, y'all are going to have to show me the awesome bubble tea shops.  ANA: That's all, Daniel. I don't know the city anymore. And I feel like he frequents a lot more shops than I do. ADRIANA: [laughs] DANIEL: Yeah, I have some recommendations.  ADRIANA: Awesome. DANIEL: We'll definitely go for some boba. ADRIANA: Yay. ANA: It's going to be like tracing San Francisco for bubble tea. We'll need a blog post. [laughs] ADRIANA: Oh my God, yes.  DANIEL: Oh yes. ADRIANA: That's right, the drink tour, the boba drink tour. I love that. DANIEL: All the spans we could collect. [laughter] ADRIANA: I love it. ANA: So, Daniel, for folks that are just getting to know you, how did you get into tech? DANIEL: So I got into tech because I didn't know what I was doing in life beforehand. I was planning on becoming an electrical engineer. And I started my first day of my master's program, and I was like, wow, this really sucks. Because I was looking at what I was doing, and I was like, I don't feel anything looking at these math equations. And I was looking at these really complex diagrams, and I was like, if I have to do this for the rest of my life, I'm probably going to be very sad.  So I decided to actually quit my master's without any plan and cold email all of the founders of the startups that I have used in the last year of my college experience. And one of them got back to me, and then I got hired the week later. So that is my journey into tech of randomness and me not liking where I was going in life. ANA: What were you studying as in bachelors that you used some tools? DANIEL: So I was doing circuits. I was figuring out how circuitry works, how semiconductors work, just very complex mathy things. Because when I first started my program, I thought electrical engineering was all robotics and cool things that you see in The Matrix, but it is just literally all math. And I was like; this is not what I signed up for. [laughter] I remember on the college website for the electrical engineering major, they have a picture of a hologram and robots. And I was like, this is what I'm signing up for. [laughter] This is so cool. But the reality kind of hit the first year, and I was like, maybe this is not for me. ADRIANA: So you wanted to make our evil robot overlords, is what you're saying. DANIEL: Exactly, but that didn't happen.  ADRIANA: Oooh. DANIEL: I'm not good enough at math for that. [laughter] ANA: I feel that. ADRIANA: I mean, honestly, it's so cool that...I think it's like a really important life skill to realize that, hey, I don't like this, and I don't want to do this.  DANIEL: Yeah, for sure. ADRIANA: I think it takes a lot of understanding of yourself and a lot of courage to be able to do that. So hats off to you for having done that and wanting to pursue something that made you happy. DANIEL: Yeah, I think I didn't have enough fear. ADRIANA: [laughs] DANIEL: Because I don't think I would have done that again if I went back in time. So I'm happy that it happened because I'm so happy with my life right now. ADRIANA: Yay. ANA: What is it that you do now?  DANIEL: So what I do now is a lot better fit with my personality. I like to think that I have a pretty fun, outgoing personality, and that's kind of what I get to do in my daily job. I get to work with a lot of customers, engineers, developers from other communities, OpenTelemetry community members like Adriana and Ana.  I get to work with all these amazing people in the community to not just sell a product but help developers understand what observability is. And that's kind of my favorite part of my job because I don't have to sell anything for my job. My job is to literally make other developers happy and help them learn and gain skills. That is like the dream job, right?  ADRIANA: Yeah.  DANIEL: Because I don't want anything from anyone. I just want to help people on their journey. And yeah, that's kind of my passion, too; I want to help developers learn and gain skills and be better engineers, which I love about my job. ADRIANA: That's awesome. I think that's such a lovely description of the type of work you do. ANA: And it's also nice when you're able to feel like your personality fits your job role. I always say bring your whole self to work. But I feel very similar, like, when I joined developer advocacy, it felt a lot more natural to me since I was already such a community-giving person and caring about other people that it was like, oh, now this is part of my job description, rad. I was going to do this anyway. ADRIANA: Yeah, I totally agree. Like, when I first heard about developer advocacy, I'm like, that's a job? Like, this is what I want to do. [laughs] Sign me up. DANIEL: Yeah, when I found out about it, too, I was like, wow, I didn't know this was a job either. So I'm so happy that I stumbled upon the job. ADRIANA: How does one stumble into a DevRel job? [laughs] ANA: Yeah. How do you stumble into DevRel? [laughs] DANIEL: This is actually a really weird, funny story. When I was in college, I had a lot of cojones. [laughter] So I didn't like...I was desperate, so I would do anything. So I actually Twitter-DMed, the CEO of GitLab.  ADRIANA: Oh my God.  [laughter] DANIEL: And I was like, "I want to come to your conference," that was happening in the city. And he responded in like 20 minutes.  ADRIANA: Whaat? DANIEL: And he was like, "Okay, here are like four tickets for you and your friends."  ADRIANA: Whaat?  [laughter] DANIEL: And then this is a core memory for me. When I went, they had this wall of bagels. Like, they had this entire wall covered with bagels that you could grab off the wall to make yourself a bagel sandwich, and I was like, this is really cool. Because that was my first tech conference I've ever been to, and I was like, wow, this is really cool.  And I heard all of these people called developer advocates or evangelists and all these engineers talk about what they're passionate about and like DevOps and really cool things that I'd never heard about. And I was like, that's a job? Where you get to have bagels and teach people about really cool technologies? And that's kind of how I stumbled upon developer advocacy. I think that was the first place where I saw a developer advocate giving a demo, and teaching, and working with the community. I was like, ooh, this sounds fun. I want to do this. ANA: [laughs] I love that you just ended up DMing a CEO of a respectable tech company. That's usually what makes a lot of people in our communities, like, you just shot your shot, and something worked out in the space. DANIEL: So, you know what's really embarrassing? Sometimes I meet people, and I realize that I DMed them like six years ago, and they didn't respond to me. So the last DM is college, me trying to get an internship. ANA: That happens to me all the time. Like, I'm hanging out with someone at a conference or, like, connect with me on LinkedIn or Twitter. And I follow them, and I go to send a message. [laughter] And it's like, 2016, they reached out for help. No answer from Ana.  ADRIANA: Oh no. ANA: And I'm like, oh, I'm sorry. [laughter] Like, some years I'm available to respond to every email, and some years I'm not. [laughter] That year, I was not. ADRIANA: Oh my God, that's too funny. DANIEL: Yes, so I was definitely a hustler when I first started out. Yeah, going back through my DM history, it's kind of embarrassing all of the DMs that I sent that got no response. But I'm glad that the people who responded did respond because they helped me, like, guide, and gave me advice to the person I am today. So I'm really happy about that. ANA: Do you think that those opportunities are still out there for folks to just randomly message people they look up to or leaders in the space? DANIEL: Yeah, for sure. I feel like also teenagers and people who are coming up in the community are a lot more savvy with social media than I was. Because I remember the first time I cold-DMed someone, it was just like, "Hi, you're so cool. [laughter] Can I meet with you?" And that's not what you're supposed to do. [laughter] But little, old innocent me was like, oh, they'll definitely respond.  I think other people who at least reach out to me come with questions that they want answered that's very pertinent to my job. Like, I've noticed that a lot of the folks that are coming up approach folks with a lot more intent and with really good questions. So I think those folks get to meet all of their heroes and get to interact with everyone in the community. So I think it's all about how you approach people that makes all the difference. ADRIANA: Yeah, absolutely. ANA: I think I read on Twitter at some point one of those advices is if you're going to cold email or cold DM someone, either be very specific about what you're asking for or put out there what you're willing to give in return, like, I'll give you this and would you mind giving me that? Whether it's an introduction or a 15-minute coffee, or feedback on anything. DANIEL: Yeah, for sure. Also, being persistent is important. Because I realized that some of these people I tried to DM probably get thousands of DMS a day. So following up in a respectful manner, I think, is also pretty important. Because I think some of the DMS I finally got responded to, I had to kind of follow up multiple times before I got someone's attention. So I think that was also really important.  ANA: Don't give up if they don't answer the first time.  DANIEL: It's probably not because they hate you because that's what I thought in the beginning.  ADRIANA: Right, right.  DANIEL: And I've seen these people's DMS, and I'm like, that is definitely not the reason. ADRIANA: They're just busy. So when they have a chance to respond back, it almost feels like you've won the lottery, right? [laughs] DANIEL: Exactly, exactly.  ANA: Or they have ADHD, open the message, forget to respond. And they totally meant to answer, but they didn't. [laughter] ADRIANA: Oh yeah, that's true. There's that angle too. I've totally had that happen to me before more than once. ANA: [laughs] So I'm just speaking from experience of a lot of my DMs that I owe people responses to before Twitter went, like, sad face. [laughter] Yeah, that whole thing [crosstalk 13:02] DANIEL: I kind of refer to that situation like, yeah, the thing over there, yeah. And it's really sad because I feel like Twitter is where I built a lot of my relationships with the community. And it feels like my home was getting wrecked, and I'm like; I don't know [laughs] because I feel like -- ANA: No, same. DANIEL: Yeah, I don't know. [laughs] ADRIANA: Yeah, yeah, it feels weird. It feels weird. For years, I resisted using Twitter. Like, [laughs] I remember when Twitter first came out, I'm like, I don't get it. Why are people telling me every minute of every day what they're doing? I don't get it. Then a friend of mine's like, "You can use it to build up your personal brand and promote your stuff." I'm like, oooh. And Twitter helped me get my current job [laughs] because Austin saw one of my blog posts that I posted on Twitter. And he's like, "Hey, how would you like to do this for a living?"  ANA: [laughs] ADRIANA: I'm like, [gasps]. So yeah, I totally agree; sad face. [laughs] ANA: Working at Google, working at Uber was technically all to Twitter, like, joined Twitter early on. Like for Google, I was able to search for...I think at that time, it was internships, and that was my in to then doing time with them. And same thing with Uber; it was the connections that I made on Latinos in tech on Twitter, it's that. Now that it's all going sad panda, it's like, but this was my roots. Like, this is what made me continue coming back, but we shall see what happens in the near future. ADRIANA: Yeah, it'll be interesting to watch as this unfolds. TBD, right? DANIEL: For sure. [laughs] I'm kind of biting my nails. But I hope that everything ends up good with that because I don't want Twitter to die either because Twitter is such a good medium. It really can't be replaced, I think. ANA: I feel like Jack made that comment when everything was going on where it's like, Twitter can't just go away. I mean, we all work in tech, and we do know that it could just go away. [laughs] But he does have a point, like, it's just not going to happen.  DANIEL: Yeah, for sure. ADRIANA: I had a question for you. So you found your way into DevRel. Is this your first DevRel role at New Relic, or did you have a DevRel role at a previous company? DANIEL: At a previous company, I was a developer advocate. But I was actually acting like a marketing person because I was the first non-engineering hire at a 20-person startup.  ADRIANA: Oh, cool. DANIEL: So that means that you're writing copy. You're trying to find contractors for the logo design, at the same time, doing documentation work, like, doing everything.  ANA: [laughs] DANIEL: So I think this is my first job where I feel like I'm doing the work of an actual developer advocate and not 16 roles in one. But yeah, even my time at New Relic, I feel like my role and the things I do have evolved over time. So when I first started, I think I was more focused on building how-to guides and getting ramped up myself on what observability is and what all of us do as observability providers.  So the first year, I was kind of working more on one-on-one content, like how to get started, what is observability. But now I think I'm more focused on enabling practitioners who are kind of in the thick of things and trying to figure out where to go from one-on-one. So that's kind of like a shift that I've done as I've learned through my job more about observability.  So I think that's been a really interesting transition for me, going from creating content geared towards very beginners that are getting started in their journey with observability compared to the audience I'm trying to target now, which is more of the folks who are running into roadblocks. They're on their journey already into getting better observability into their systems. ADRIANA: Cool. And as part of your observability journey, did you get...like, you're active in the OpenTelemetry community. Did you start working on OpenTelemetry when you joined New Relic, or did that come about a little bit later? Like, how did you get in with the community?  DANIEL: I think the first time I ran into OpenTelemetry was because one of our proprietary solutions couldn't do what I was trying to do. So I was trying to trace a particular application that did not have support from our language agent. And instead of trying to set up a 45-minute meeting with engineering managers and getting something on the roadmap...that stuff is very stressful. So instead, because we support OTLP and OpenTelemetry natively in our platform, I decided to try tracing this system with OpenTelemetry because someone built a collector receiver for this particular application. And then it worked out of the box.  ANA: [laughs] DANIEL: Like, it took me 20 minutes to set it up, and I was like, oh my God, this is amazing. I didn't have to set up a meeting to get this type of data into New Relic. ANA: [laughs] ADRIANA: Magic. DANIEL: And that's when I really realized the power of community and open source because you don't need to rely on a 10-person team to maintain all of your software, so if something breaks, you have to go to that team. With an open-source ecosystem, you can pull down existing pieces of code, edit it to fit your needs, and then just continue on with your day.  And that's what I love about OpenTelemetry is that there are so many contributors all over the world that are building solutions for their use case and sharing it upstream to all of the users out there, like what I did with this particular receiver. And yeah, that's what I found amazing about OpenTelemetry, and that's why I got more involved in kind of being part of that community. ANA: Nice. And it's also that where it's like, once you start with one project, you kind of realize how big the open-source community is. And I can speak for the Cloud Native CNCF projects, also, where it's just like, once you get into one, everything kind of starts making a little bit more sense, or there's always room for collaborations with other projects. DANIEL: Oh, definitely. And something else that I think I've been really passionate about in my role is evangelizing observability in communities where observability isn't the main thing that they're concerned about. Because when I go to CNCF events like Observability Day, people who are going to Observability Day already have a pretty good grasp on their observability strategy. Like, you wouldn't be showing up unless you had some kind of vested interest in observability.  But when I talk to a lot of my friends who are developers, observability is something that they think about in the back of their mind. It's never in the forefront. So what I've been really passionate about in the last couple of years that I've been doing this is bringing up the topic of observability to audiences where it's not their primary focus. So I've been working a lot with the awesome folks from the Kafka community over the last year because a lot of the Kafka operators I talk to at conferences are virtually...they're just worried about keeping their systems up. ANA: [laughs] Yes. DANIEL: Observability is like, if your couch is on fire, you're probably not trying to rearrange your table. [laughs] You're trying to put out the fire first. [laughter] So when I go to these conferences, a lot of the talks were about optimizing their setup, performance engineering, things that were very related, but it wasn't about observability. It's more about how do we maintain a system that works?  So I really want to go to these communities and kind of expose...just hold up a sign that says "Observability" so more people can at least start thinking about it and implementing solutions for their setup, like, for example, using OpenTelemetry to implement tracing or Prometheus for gathering metrics from their Kafka environments. So yeah, that's what I've been really passionate about is, talking about observability to folks who should care, but they're probably so overloaded with just keeping the lights on that it's probably not in their top 10 list. And that's been really rewarding for me. ANA: That's true. There are a lot of those communities out there. I think Kafka is a perfect one. You mentioned a year of working with them. [laughs] I think it's very continuous work because due to the organizations that end up using their services, it is such a critical point of a software, of an application. You really have to make sure to understand it to be able to make it reliable.  Like, I know at the last company, when we would talk to a lot of our customers, it was like they were constantly afraid of Kafka, Cassandra. Those two are always on the top five list of what tool within my toolchain is it that has had a lot of incidents, or we're constantly firefighting? And how is it that we can get ahead of the curve? DANIEL: Yeah, for sure. Kafka is like...that's the reason that I speak to a lot of the folks in the community because observability really will help them in the future get a better position, better provision resources for their future needs. And something that I've been really passionate about also is leveraging my internal company innovations and sharing them with the world. Like, we have a team internally at New Relic that over 100 people are working just on maintaining our Kafka instances. And out of that team, there are experts, like, world experts on Kafka and monitoring.  ANA: [laughs] DANIEL: So I've been working with them to try to share some of the knowledge that we are learning internally operating at such a scale to the rest of the community. So that's also something that I've been really passionate about. ADRIANA: That's super cool.  ANA: Is some of this content already available online? DANIEL: Yeah. So I will link it later. [laughs] I will share it with you guys later. ANA: [laughs] Yeah, we'll put it in the show notes.  ADRIANA: Okay, cool. DANIEL: Yeah. I recently published...well, not recently, like middle of the year. I published a guide on how to do distributed tracing with Kafka using OpenTelemetry, and this is something that we implemented in some of our internal systems to start getting better visibility into our own Kafka instances as well as implementing Prometheus, like, which Prometheus metrics you should pay attention to in your Kafka environment.  And we did a talk on that at the Kafka Summit in Austin. So yeah, I'll put it in the show notes. There are a couple of really cool pieces of content that me and a couple of engineers that are experts on Kafka we put out to help folks who want to get started in their observability journey with Kafka how they can get started with OpenTelemetry and Prometheus. ADRIANA: That's awesome. I have a question for you around that. As far as the stuff that you choose to surface as a DevRel, is it stuff that you choose to surface because it's like, hey, this is something that I see people are struggling with, or I've just learned about it, and therefore I want to write about it? Or is it something where you're like...or someone tells you, "You should take this direction. You should write about this. You should dig into that." DANIEL: I don't think I've ever been told in my time at New Relic what to write, thankfully. It's usually me struggling myself with something and then me writing about it because I had such a hard time. I think when I talk to folks, I say my superpower is not being the smartest person in the room. I am not the best engineer, which makes me such a good person to have because in a team of very smart engineers, you always need that person that will ask the stupid, obvious questions, and I am that person. [laughter] So I am in these fancy-schmancy engineering meetings, and I'm like, "What is A?" And everyone's like, "Oh," and then they take the time to explain it to me. And I try to share that knowledge that I gather too, like, share with the community because since I was little, I never had any shame asking questions. You know how people are embarrassed to ask stupid questions? I would raise my hand high. Like, I did not have hesitation to ask a stupid question.  ANA: [laughs] ADRIANA: That is awesome.  DANIEL: And I think that's one of the greatest gifts I have is not having shame to ask questions because I always get people afterwards after a big meeting being like, "Oh, I was wondering the same thing, but I just didn't know how to ask." So I think that's something that allowed me to be really successful at my job. That's how I find content is figuring out what I struggle with and trying to share it with the community.  ADRIANA: That's awesome. I think that's such a good skill to have because, like you said...I remember even early in my career; I'd just sit there cowering in silence like, I have a dumb question I don't want to ask. I'm so confused. I'm so lost. And then, at some point in my career, I'm like, I don't care anymore.  ANA: [laughs] ADRIANA: I'm going to ask it because I'm confused and lost, and I need to know this information to do my job, so screw it. I commend you for having the cojones to ask those questions and make it safe, like, make other people feel safe for asking those questions as well. Because I think when you hear someone asking, quote, unquote, "stupid questions," it empowers other people to also ask the, quote, unquote, "stupid questions," which are not stupid because it's obviously knowledge that people are seeking. ANA: I know I was always the person that was afraid of asking questions for multiple reasons. [laughs] So when people are like, "I don't have any fear of asking questions," I'm like, "Oh, here are my questions. Can you ask these?" It's gotten better.  ADRIANA: [laughs] DANIEL: Oh, other people have done that too. Honestly, some people or some folks will DM me things to ask, which I'm happy to do.  ADRIANA: That's awesome.  DANIEL: And I think that's also a really good way to pressure test your documentation. Sometimes what I do when I audit some documentation, I screen-record myself, and I ask all the things that pop into my head as I'm going through it. And that's a really good way to make your documentation and things that are in your product a lot more accessible is having a person go through it and just stream of consciousness, like, say everything they think about their experience. So that's something that I've used internally as well to improve our product. ANA: That's actually a really cool activity, whether you're in DevRel or not. Any engineer kind of thinks about the work that they do because, I mean, even when you build internal tooling, you still have other teams that are depending on your tooling, so you do end up writing docs. And I think sometimes a lot of engineers don't think about what the actual experience is when they use the software that they're building. Maybe they should be doing that more. DANIEL: Yeah, more documentation is always better. And yeah, I've been super thankful that the documentation team I work with internally has been so open to feedback that I'm able to get a lot of changes implemented very quickly for the things that I struggle with. And I have a saying that goes if I work here and get paid by my company and I don't understand it, the likelihood of someone else that has nothing to do with the company understanding is probably even lower. So that's how I like to justify it to myself when I add 45 comments to a Google Doc about documentation. That's what I tell myself. [laughs] ANA: I mean, I always see developer advocacy, developer relations as user zero. Like, you are at the start of that journey. Like, we have to test everything before actual users do get to touch it, prospects or customers or community. ADRIANA: Yeah, we're the bridge. I do feel like documentation in tech that's something that the whole industry struggles with, especially open source. Like, I cannot tell you how many blog posts or technical docs from whatever open-source project I've read where I'm going through, and they're like, oh yes, just do blah, blah, blah. And I'm like, okay, but like, how? [laughs] Like, am I the only person who has this question? It's not possible.  And then you're like Googling all this shit in the middle of the night trying to figure out, [laughs] like, has someone else had the same question as me? Please, God, let it be that someone else had the same question as me. So I make it my mission to ensure that whenever I write technical blog posts, I include the excruciating details because I just freaking hate it when people leave those things out.  Because I feel like engineers writing documentation partway through, it's like, I'm bored. I don't want to do this anymore, byee. We need to learn to not do that as a tech community in general. I'm not saying for DevRels. I'm saying anybody writing technical documentation has to be better about filling in the blanks, right? DANIEL: For sure. I think accessibility is really, really important when it comes to onboarding new people to your open-source project or a product. And doing these studies where you look at how people interact with your developer assets, whether it's documentation, or a demo, or anything, it's really important to make sure as many people who want to try your product, or open-source project are able to actually go through with it. Because if I get stuck in step two, the likelihood of me figuring out the problem and getting back to step three is very low. Like, if something takes me more than 30 minutes to spin up a getting started thing, I probably will abandon it. ANA: Developer happiness where you're just like...it's like, I mean -- ADRIANA: Yeah, that's fair. ANA: Sadly, it's that too, there are so many shiny things out in the tech space or so many competitor tools to one another that it does become like, what is the user experience of using this open-source tool? Of these three vendors, what is actually easy for me to implement? And was it well-documented? Is their customer success team getting back to me in a timely manner where I can do my job and report to my leadership team the work that I'm doing? I think we have seen in the industry, like, I think in the last three, four years, there's been a higher growth of caring about developer happiness, developer experience. DANIEL: Yeah, for sure. I am so happy that the industry as a whole is moving towards the idea that we want to make the experience for developers as smooth as possible. Because I've noticed that even internally, we prioritize ending developer friction over a lot of other OKRs because we want to make sure that people are happy using our product because when it comes to renewal time, all the other companies are going to be like, "Well, ours is better, and your developers won't be suffering as much." And that's like a huge selling point for a lot of products, especially developer-oriented products. ANA: I even want to go a step further and say when you are working in the SRE space, it is just more niche, and it kind of matters a little bit more since you kind of need to maintain all the systems up. So people are having more scrutiny, I think is the English word for it, like just microscope, like looking at all these little details. DANIEL: For sure, for sure. And another aspect I think that's really interesting for me is how we value interacting with developers. Before developer relations really became a thing, I feel like there was kind of a disconnect between companies that made developer tooling and developers themselves. And I've noticed a trend where in the last couple of years, companies are hiring roles like developer relations, or they're having very intentional community programs where they're constantly interacting with the users of their product to continue to iterate, and I'm really happy about that.  Because when I was first getting started with tech, like when I was in middle school and high school, and that wasn't even that long ago, companies didn't have community programs like they do now where they would answer questions from someone that wasn't paying them. The fact that there are free tiers for a lot of companies that are product-led growth, I think, is amazing. And it's going to make everyone better because competition, I think, increases the quality of all the products that are in the playing field. ADRIANA: Yeah, totally. I think one of the important things as a DevRel is to listen to the feedback from the community because it's one thing to be a talking head and to stand on your soapbox and say, "Thou shalt do it this way," and then it's another to get feedback from your customers. Like, hey, buddy, this isn't at all how it works in real life. You got to get back to reality here with the rest of us. And I think it's so important for DevRels to be in touch with the real-life scenarios that are happening in the community so that we can relay it back to our employers or to the open-source projects that we work in so that we can increase that developer happiness. ANA: Most definitely. One of the questions that I had for you, Daniel, is that I also know that you have a project called Bit Project; you're like the founder of it to get some folks started into DevOps. Can you tell us a little bit more about it? DANIEL: Yeah. When I was in college, and I was in electrical engineering, I was already starting to think about making a pivot to software engineering. So I started to teach myself coding. And I started a school club within my college where a bunch of friends would get some pizza or some boba, and we would congregate in a meeting room. And we would just teach each other things that we learned that week. And it started as that, and then we kind of grew it into a student organization where we built really fun, quirky ways to teach basic tech concepts because I'm a fun, quirky person, and I like to teach things in a fun, quirky way.  ANA: [laughs] DANIEL: So me and my friends built awesome, fun, interactive ways to learn different things. So one of the things that we built was a way for you to play Tetris, and it's more like...it's not Tetris but it's more like...I don't know how to describe it. So there's basically a giant grid that people could send cURL requests to that particular website where you could change the color of a particular grid, like a square within the grid with a POST request.  And we taught people how to use Postman, and how to use APIs and cURL, and how to send requests, what HTTP requests are, things like that through a game. And that's kind of like the essence of what Bit Project does. We try to teach people complex things using fun and quirky ways. So we have had people build serverless-based projects where people built like a detector where if you could send an image to the website, it'll tell you what song would match your particular face. We had students just build really fun, quirky projects with cool technologies that aren't really exposed in school.  I'm super excited. We're working on some really cool projects for next year, teaching students about serverless and APIs and all the cool technologies that are out there. That's what Bit Project is essentially. And I'm so happy that that's how I got my start because I get to kind of translate that over into enterprise software, where I think you would all agree that we could use a little more fun and quirkiness in this space. ANA: [laughs] ADRIANA: Yes, I completely agree. ANA: I mean, it's that, like, the stuff that we learn in school is not necessarily what we apply actually on the job. So when there are programs that help people filter out what is actually taught in school to be applicable, that's been my story. And it's also the work that I do. I remember when I first saw Bit Project was the SRE stuff that you were working on, I think, two years ago, and I was like, ooh, that's super cool. DANIEL: Yeah, I'm super excited also that there are other programs in high schools that are starting to expose students to tech concepts early. These are very select schools across the country. It's not like everywhere. But there are certain schools that are exposing students to more realistic avenues of coding, not just designing number counters but actually building websites or building tools that engineers in the real world would use. So I'm so happy that there's still a movement out there that's growing where we're starting to expose students to the real-life idea of a software engineer and what people in our industry actually do. ANA: And on that one, I'm going to plug that there was one by Major League Hacking that they were working on last year. That was an SRE one as well. It was being built; I think, paired with Facebook production engineering team. So it was kind of like really taking those concepts to give them to college students out anywhere in the...I think it's mostly North America that Major League Hacking works with. I'm also a huge fan of seeing that work be built out because it's needed. Like, it's really hard to catch the difference sometimes of what you are learning in school versus what's actually applied. ADRIANA: Yeah, and they don't teach you how to use Linux in school, let me tell you. [laughter] ANA: What is a server? [laughs] ADRIANA: I agree. I remember when I started university, the first day in my computer class, we're like at a Sun terminal, and I'm like, what the hell is this? I'd never seen Unix in my life. I'm like, what's going on? I didn't even know there was anything besides Windows. And then they just sort of expect you to know this stuff.  So the question I had for you was like, you know, it's awesome that there are these programs out there that bridge that gap between the theoretical and the practical. But does that make you wonder should schools be doing more to bridge that gap themselves rather than having to rely on these satellite programs to bridge that gap? Because I feel like that's still missing. I want to get your thoughts on that. DANIEL: Definitely. The problem is that academia is, I think, particularly slow to adapt curriculum. And for a thing like maybe law or biology, where the basics stay the same, it's probably less of an issue. But with something like computer science and software engineering, the goalpost moves every day. Like, there's some more level of abstraction that developers have access to. Like, if you compare what a software engineer does now compared to 30 years ago, it is extremely difficult.  ADRIANA: Totally. DANIEL: So even though schools should be adapting to the times, the times are moving so fast that I don't think it's feasible for a lot of the schools to adapt their curriculum to the rate of innovation that we're seeing in our industry. So I think it's important for schools to build in flexibility and real-world engineering into their curriculum. And I think universities like the University of Waterloo does a really good job at that because a core part of their curriculum is students going into actual companies and internships. I don't know what they call it. I don't think they call them internships.  ANA: Co-op. DANIEL: Co-ops, yeah. They actually have to work with real-life systems and real-world problems, and that's I think why Waterloo grads I think are the cream of the crop when it comes to getting their first PR in the fastest. ANA: [laughs] I mean, it goes to that where it's like, who's the most comfortable touching tools, touching software, and not being afraid to ask questions? And a lot of it does translate to what have they been exposed to when they're learning. I work with a foundation called National Academy Foundation. They partner with high schools to academies like on verticals, and there's one for IT or networking and engineering. So they learn and by senior year of high school, they've interned at one or two places for networking, or development, or graphic design, and they get certifications.  So I think the earlier we're able to expose students to whatever career they might be, like, I'm not talking about just doing this for tech. I feel like it's a lot easier for folks to commit to going to university for, like, this is what I want to do when I grow up, and also be aligned with this is how much salary I could be making if I do study this. I think prior generations didn't really have access to that perspective. DANIEL: Yeah, for sure.  ADRIANA: Yeah, that's true. DANIEL: And I think it's moving towards the better. Like with the internet, with TikTok, the rate at which students get exposed to new ideas is quite fast. And there are so many creators out there that are advertising all the opportunities that are available in tech and in IT. So I'm hoping that we get more diverse and cool candidates that are coming through the pipeline and be the future leaders in our field. I'm so excited for that. ANA: I 100% agree with that. As we're getting ready to close out this episode, I wanted to ask what advice do you have to folks starting out their careers in tech? DANIEL: I think not giving up is a really, really important piece of advice because right now, especially, it is the worst time to look for a job, especially someone with not a lot of experience because there are so many people in the market. And then there are not enough jobs to go around because of the economic situation.  So don't give up if you don't find a job in your first two months or three months or even a year of looking because it's probably not you. It's just that there are so many candidates and applicants for particular jobs. It's so competitive that it's more than likely that you're not going to get the job in your first couple...even a couple hundred times that you apply. So I think not giving up is really important.  And also, being part of the community is really important. A lot of the jobs that I've gotten in my career have been through connections I've made both virtually and in person. So creating friendships, being part of the community, I think is a really, really important differentiator when it comes to separating yourself from the 4,000 other applicants for the same job that you're trying to go for. So I think building up a personal brand, being part of a community, having real-life friends in the community is really, really important. ADRIANA: I think that's really awesome advice, and I could not agree with you more on that. ANA: I plus-one all of that. I feel like personal brand also definitely really helped my career and not just transitioning to DevRel but just as a strong engineer. ADRIANA: Yeah, absolutely. ANA: Well, with that, thank you so much, Daniel, for joining us in today's podcast. We loved talking to you about your start into tech, observability, Bit Project, and all the cool things. Don't forget to subscribe and give us a shout-out on all social media via oncallmemaybe. We just joined Mastodon not too long ago so check us out there and be sure to check out the show notes on oncallmemaybe.com for additional resources and to connect with us and our guests on social media. For On-Call Me Maybe, we're your hosts Ana Margarita Medina... ADRIANA: And Adriana Villela signing off with... DANIEL: Code, peace, love, you know, or whatever combination thereof. [laughter] I hope everyone has a great time and you're not yelling at your code today.  ADRIANA: I like your spicy take. [laughs] ANA: He's like, I'm Daniel. There's no such thing as peace, love, code. [laughs] ADRIANA: You just turned it on its head. It's fantastic. ANA: Randomized string. ADRIANA: That's hilarious. [laughs]

About the guest: Jen Shute Benson (she/her/hers) is a Senior Director of Platform Engineering at Slalom Build. She has worked for over 25 years in the tech industry. Her early career included quality engineering, application support engineering, and systems engineering. She moved into leadership and has managed Cloud Infrastructure, Cloud Solutions, DevOps, and Platform Engineering teams. She has managed two major cloud migrations at large-scale enterprises before moving into the world of consulting. At Slalom Build, in addition to leading Platform Engineering teams in 3 geographic locations, she supports multiple markets and clients and is heavily involved in ID&E initiatives within her Platform Engineering capability and at the Slalom Build level.  Jen also volunteers as a mentor for Slalom’s Women Who Build program. She is a member of Cloud Girls and co-chairs the membership committee. Cloud Girls is a non-profit organization dedicated to community building and celebrating the success of women in cloud careers, and giving back through awards and charitable programs. Find our guest on: LinkedIn Find us on: On-Call Me Maybe Podcast Twitter On-Call Me Maybe Podcast LinkedIn Page On-Call Me Maybe Podcast Mastodon On-Call Me Maybe Podcast Instagram On-Call Me Maybe TikTok On Call Me Maybe Podcast YouTube Channel Adriana’s Twitter Adriana’s Mastodon Adriana’s LinkedIn Adriana’s Instagram Ana’s Twitter Ana's Mastodon Ana’s LinkedIn Ana's Instagram Show Links: HugOps Slalom Build HRIS Microsoft Access Quality Engineer (QE) Dreamweaver Heroku Active Listening Cloud Girls Additional Links: Slalom Women Who Build Program Transcript: ANA: Hey, y'all. Welcome to On-Call Me Maybe, the podcast about DevOps, SRE, observability principles, on-call, and everything in between. I am your host, Ana Margarita Medina, and with my awesome co-host... ADRIANA: Adriana Villela. ANA: Today we're talking to Jen Shute, who is a Senior Director of Platform Engineering at Slalom Build. Thank you for joining us today. JEN: Thank you for having me. I'm very excited to be here with you today. ANA: In true On-Call Me Maybe fashion, we always like asking our guests, what are you drinking today?  JEN: Okay. So I've brought with me some kombucha. It's organic brew, Dr. Kombucha, and it's a new flavor that I've never tried before. It is blood orange ginger, and it is very good. I highly recommend it to anybody who likes kombucha. ANA: I'm a huge kombucha nerd, and it makes me want to have one of those this morning too. That ginger sounds delicious. What about you, Adriana? ADRIANA: Today I've got water.  ANA: Nice. ADRIANA: Nice and simple, my go-to drink. [laughter] How about you, Ana? ANA: I'm going to make some water and some caffeine. So if you've listened before, it's usually the yerba mate with mint. I'm doing one of those today; just trying to pick up the energies. With all these California storms going on, I'm like; I need all the positivity energies that I can get. ADRIANA: Oh yeah, you guys finally got a break in the weather. Somewhat. [laughs] ANA: We had a 24-hour break, and we're back to another three, four-day storm.  ADRIANA: Oooh. ANA: So it's like, I think, four atmospheric rivers hitting California.  ADRIANA: Holy cow.  ANA: And for folks that are interested, they should look at the maps of what it was like the drought in California in December and what it's looking like now and the fact that some parts of California are still in drought and a lot of parts are very much flooded. It's crazy. Global warming is insane. ADRIANA: Yeah, well, definitely a stark contrast to Toronto. We actually got snow today. I woke up with, I'd say, five centimeters of snow. What's that in inches? Like two and a half inches. So yeah, we got a dose of proper winter after rain for the last couple of weeks, so yay. [laughter] Definitely better than the floods. I'm sorry you're having to go through that. That does sound very awful. So I hope it subsides soonishly. ANA: Thank you. Definitely HugOps to California. I feel bad for all of it. And, I mean, I think the entire world has been having some crazy weather right now, and hopefully, it is a bigger wake-up call to those that are currently not looking at it or being proactive about the actions that they take against the environment. ADRIANA: Yeah, I totally agree. What's the weather like in your neck of the woods, Jen? JEN: I'm in Ohio. And I think I generally I'm pretty close to what you have in Toronto. You know, I haven't been outside today. I just kind of peeked out the window to see if we got any snow since you mentioned you had snow. It doesn't look like it. But I think we're probably in pretty similar climatic regions, so we both got a little lake-effect snow and -- ADRIANA: Yeah, totally. ANA: I like snow. I would not be able to live somewhere with snow. I think this rain made me realize -- ADRIANA: [laughs] ANA: This type of weather just makes me extra sad, and that's really hard.  ADRIANA: Yeah, that's true. ANA: So with snow, I would not be able to live, like, I've always lived tropical or next to water. I think California is the coldest I've ever lived, which says a lot. [laughs] ADRIANA: Oh dang. Oh yeah, that's right. You lived many years in Miami, right? ANA: Yeah, I've lived in Costa Rica, Nicaragua, Miami, and now San Francisco Bay area. So pretty privileged, I would say. ADRIANA: Nice, warm weather. I do miss that. I do miss that. [laughs] ANA: Well, Jen, since we have you here, we wanted to hear how you found your way into technology. What was your path like from that moment when you were like, this was cool too. I can make money.  JEN: I actually had a really roundabout path, and I think that's not uncommon for people in tech. In college, I took psychology, so I graduated with a psych degree. I worked in human services for a couple of years. I found that I was not making enough money, unfortunately, to support myself. So I tried to figure out what's a transition I can do? Like, I don't feel like I'm ready to, you know, or I'm at a point where I could go back to school. And I thought human services might be a shift that I can make a little easier.  And so I got a job in HR. At the time, it was myself, and there were three other women that I worked with in our team. And I was the only person, you know, I was in my 20s at the time. I was the only younger person. And we had a HRIS where we kept all of our HR data, and nobody else really wanted to use it. And so I kind of became the person that took over that HRIS, and I did all the management. And it kind of got to the point where if we needed to be tracking something that we weren't...I'm kind of aging myself, but it was an Access back end. I learned -- ADRIANA: Oooh. Good times. I remember Access.  [laughter] ANA: Same. I'm certified in Access 2005. [laughter]  ADRIANA: Oh my God.  [laughter] JEN: So I learned how to go in and add fields and did a couple of trainings at the company that created the HRIS that we were using, and I actually ended up getting told about a QE position they had there. And I had always kind of thought that people who worked in tech were super cool. I always thought that's something I could never do. But I went ahead, and I applied for the job, and I actually got the job. So that's how I got into tech.  So I worked in QE for probably about five years. And then I switched over into systems engineering and then kind of worked my way through. And as things changed and we started moving to the cloud, I started working more in the DevOps realm and cloud. So that's my journey so kind of a roundabout way, but it got me here. ADRIANA: That's such a cool path to tech. And what are the things that you're like, oh my God, so much has changed in tech from when I started to where I'm at now? What do you think is the biggest thing where you're like, holy cow, I could have never envisioned this happening tech-wise? JEN: So this is a great question. And I feel like it's a hard one to answer because I would love to say everything. Like, when I first started, I was testing software for a client-server application. And when I moved into systems and infrastructure support, I was supporting a monolithic application that was on-prem. And what we're looking at is scalable systems that are resilient, and self-healing, and containerized. I almost have to say, like, everything has changed, and I would not have predicted any of that. ADRIANA: Yeah, that is so true. I kind of feel the same way. I mean, my first foray into programming was on a computer that wasn't even on a network. There was no internet that I was aware of at the time. I think it existed at the time, but I wasn't aware of it. Writing code in QBasic, [laughter] like, I don't even think QBasic exists anymore. [laughs] ANA: Or even looking back at the text editors we were using, like, I was using some obscure one.  ADRIANA: Yes. ANA: And then I ended up using Dreamweaver 8. And to think I was doing more of like websites and stuff, so using FTP to upload sites versus just going into the server in other ways. Like, it's interesting to look back on things like it. JEN: I mean, it really makes you wonder, like, what's next that we can't even imagine yet. How advanced are we going to be when we're going to be looking back on this and be like, "Oh my God, remember when we were in AWS?" ADRIANA: Yeah, it's true. And as radical a change as it is from starting out in one's career versus now, at the same time, I think everything's just been slowly evolving towards where it is. I was having a conversation with my dad the other day, and he was like, "Yeah, I was doing stuff in the cloud before it was even a thing." ANA: [laughs] ADRIANA: And he was talking about, "Yeah, I was using Heroku." And I'm like, oh damn, yeah, I totally forgot about Heroku. Stuff like that has paved the way for where we are right now, which is super trippy. ANA: It is always interesting to see when Heroku comes up in conversations. ADRIANA: [laughs] ANA: Because everyone literally always forgets about it. It had always done so much movement around empowering developers. So it's interesting how developers, like, it was one of their first aha moments in this type of platform space. But because other vendors have such larger marketing budgets, you kind of put it somewhere else in your brain, and like, who taught you how it works? You forgot about it. It's interesting. So with the current job that you do right now, Jen, what is it like managing the staff that you have in your day-to-day? JEN: I mean, first of all, I have an absolutely fantastic job. I really have found a home at Slalom, and so I love what I do. My role is pretty diverse. I do people management; I do sales support, I do client relationship management, project oversight. The thing that I really like most about it is I'm constantly learning. This is my first time that I've worked in consulting in that world at all. And so that's even been, like, over the last three years, such a growth opportunity for me. So it's great. It's busy. It's learning every day.  ADRIANA: So at what point in your career did you...it sounds like you were more hands-on with the coding earlier in your tech career. And now you're, I guess, a little more hands-off, and correct me if I'm wrong. At what point did that transition for you? And how was that transition for you because it's not always for everyone either, right? JEN: It was gradual. I worked as a, like I said, systems engineer for a number of years. And I moved into my first management role managing a small team. I'm going to say it was probably ten years ago, although if I looked at my resume, I might be off, you know, plus or minus three years, you know, initially managing small teams. I really was still able to get hands on keyboard and help my team with troubleshooting issues.  As my responsibilities grew, I kind of moved into one of those roles where you're in meetings most of your day, and there were things that I liked about it and things that I didn't. I do miss feeling like I could just jump in and grab a keyboard and help my team out when they need help. And there are times I think, man, I really want to go back and do that again.  One of the things that I love most about my job is being a coach and mentor and helping other people grow. And I think, too, that probably also comes from that psychology background. And my career choice was really to go into some kind of counseling, and so it's kind of taking that and implementing it in the role that I'm in right now. There are times I miss it, but I also really love what I'm doing right now as well. And given the choice, I probably would choose to do both, but I like what I'm doing right now. I wouldn't give it up, probably to go back in time. ADRIANA: Cool. That's awesome. And it's really cool also having the psychology background to lean on to help you with being a people manager because dealing with people is friggin hard. [laughs] JEN: Exactly, yeah. It sure can be.  ANA: [laughs] And it's a part of that...because I think there are a lot of people that go into management just because, one, they feel like they need to in order to have a career progression. And then there are some people that go into management because a company needed managers, and they got thrown to be a manager. So when you do find managers that, one, want to be people-first managers but, two, find such huge passion and have the skill set to do it, it is kind of like those unicorn managers that are really hard to come by. Because it's that, like, once you make people happy, believe in themselves, that's when they can really bring their best selves to work and do their best job. ADRIANA: Yeah, and really being able to mentor them is, I think, key. I mean, I think both Ana and I we wouldn't be where we are if we didn't have good mentor managers that helped us along the way. So knowing that you can shape someone's life in a positive manner is super awesome. JEN: Yeah. And I have to say, I've been really lucky, too, because I think a lot of times the way you learn how to be a good mentor and a good manager is by having good mentors and good managers. Some of my managers that I've worked for in the past that I can point to and I can say you know what? I really grew under that person, and I learned to be more confident and I learned how to take care of my people. And so I guess it's also kind of passing on what had been given to me in the past. ANA: What would you say are two or three things that you would pass on to folks that are wanting to become managers, people that want to be people-first managers or those managers that are trying to be more human-focused, or those VPs that are like, I should probably care about all my staff a little bit more, but I'm so overwhelmed with work? JEN: So to think about what are the things that I think I would tell someone to be a really good manager, or mentor, coach, give feedback. Give kind feedback, but give feedback and give it timely. And try and give good, specific feedback and help that person with how they could do something better. So I think feedback is key.  And then I think the other thing is having empathy and listening and hearing what someone is telling you. I've actually recently even myself I've been working on active listening, and it's a real challenge because a lot of times, I want to reach out and relate to that person and connect by saying, "Oh yeah, me too. I had this similar..." But I've been going through a one-year leadership program at my current company, and we've really been focusing on active listening. And it's really been a big change for me to see the benefits that come out of that. ANA: For listeners that don't know what active listening is, can you describe it? JEN: Sure. Active listening would be listening to what someone's telling you. Ask clarifying questions. Try not to feel the need to jump in and save the person. And this may sound contrary because I just mentioned giving feedback, so I don't want to say that you should not give feedback. But in this situation, when you're active listening, just be a listener and try not to interject and give your own personal experiences and just let someone else talk. ADRIANA: That reminds me a lot of therapy sessions where [laughter] with your therapist, they're listening, and you're doing all the talking, which it's a technique. I'm sure [laughs] you're well aware of this. But yeah, that's what came to mind when you mentioned that. JEN: You know, it's funny because I think being a manager, you are a therapist sometimes. That's what you're doing. ANA: Yeah. Not to derail us off topic, and at the same time, as an SRE, you end up being a therapist for systems. You're listening in. ADRIANA: Oh my God, yes. [laughs] It's so true. ANA: For all of our listeners, I'm sure you all -- ADRIANA: This is the hot take of the day. [laughs] ANA: The cross between reliability and mental health is where Ana's brain lives up 87% of the time.  [laughter] ADRIANA: Actually, going back on the SRE thread, one thing that's near and dear to a lot of SREs' hearts is on-call. [laughs] And I think a lot of people have very visceral reactions to on-call. I was wondering if you could talk a little bit about...we talked in the pre-chat you've had experience with both being on-call and managing on-call. So I think it'd be really interesting to have you share those two viewpoints.  JEN: For me, being on-call, especially early career as a systems engineer, was honestly terrifying because I was so afraid that something was going to come to me and I wouldn't be able to solve it. And so it was very terrifying initially. The more I adjusted and got used to it...I think the weeks that I was on-call, I was part of rotating on-call systems where I'd be a week off, two or three weeks back on again.  I didn't really sleep well the whole week because even if I wasn't getting an alert that I had to get up and respond to, I always knew that there was the chance that I would. Or I'd wake up in the night, and I'd grab my phone and look at it because I'd be like, oh my gosh, did I miss something? Had I slept through? So it was definitely stressful. And I think there is a level of burnout that can come if you don't take care of yourself when you're on-call. As a manager, one of the positions I was at when I was on-call, the company would...when we'd have an outage; basically, anybody who managed a team that had a system that even might remotely touch the system that was down would get on a call. I think that was pretty exhausting as well because there were some months where I was weekly getting up and getting on a call in the night. And there may be a couple of things I might have had to contribute, but I may have just even been a listener. So that was also an exhausting experience for me. ANA: I think we need to have those conversations more. I think they've happened more recently, but that burnout of being next to on-call or just even acknowledging that burnout can happen, having that conversation, like, what does burnout look like? What are ways to prevent it? And what do you do when you are nearing burnout or if you see a colleague being burnt out? Because burnout is maybe sometimes a signal to sliding into depression. Or it just shows that something in an organization is not working well. But we have not been trained in our jobs to be on the lookout for those things, that psychological safety kind of extending a little bit more. JEN: And I think, maybe I'm making a generalization here, but I think as women, we're often raised to take care of other people but not to take care of ourselves. And I will own that because, as a manager, I felt like I did a lot better job of taking care of my team if they were up in the night resolving something. I'd be very clear to tell them like, "Hey, sleep in tomorrow. Take some time back." It wasn't something that was as easy to do to take care of myself.  And I think that's something I've had to work on for myself, maybe even over the last ten years. I'm maybe getting better at it. And that's, you know when you're on the plane and they say to put the mask on yourself so that way you can take care of someone else. But that's something I've been working on, but I think it's something really important for people who are in those types of roles to know it's okay to take care of yourself. It's okay to say, "Hey, this is what I need." ANA: Leading by example when it relates to mental health in our development, technology spaces, and diversity as well is what makes the most sense. As you mentioned that and leading by example, what tips do you have for managers to actually protect their on-call teams' mental health? JEN: Being aware and making sure they're giving people time back but also having constant, like, open dialogue and checking in with people. Do regular one on ones with your team. And I know this is something that we all think we should do, and I know some managers don't do that. But have those one-on-ones and check on people. And "What do you need?" And when someone tells you, "This is what I need," be responsive to that. ADRIANA: Yeah, I totally agree. On the topic of one-on-ones, in my last role, I was managing a team of, I think, 13 to 15 people. And I had made a point of doing regular one on ones. And it's very mentally exhausting. But also, I know that my direct reports really benefited from it. And it was cool because I tried to provide a safe space for them to talk.  And so oftentimes, it's like the therapy thing. I'd have people telling me about some of the challenges in their personal lives. And part of my job was making sure that I can still support them so that they feel like they can still perform at work and deal with their personal problems, which I think is so, so important. JEN: That's so hard too. When you have 13 to 15 people, that's a lot of people to be responsible for, and to have those weekly one-on-ones, I mean, that's a whole day a week. So it is really hard when you have that many people reporting to you. ANA: Apart from that, do you ever spend any time looking at how often every single team member is getting paged, like the hours or things like it? JEN: In my current role, because it's consulting, I don't have any team members that are on-call. However, this is something that in previous roles where I had team members that got called when they were on-call, that was something that I would try and keep my eye on. And, like I said, I did try to be really proactive about if somebody was spending a lot of time off standard work hours dealing with things to make sure that they were getting time back and that they had what they needed. ANA: I was asking because that was something that I actually felt and have seen in the industry a lot where I totally burnt out of being an SRE at Uber for various reasons. And one of it was not getting training to be on-call but getting just thrown into the rotation to having just so many pings and getting woken up in the middle of the night, being expected back at the office at 8:00 or 9:00, which it's just not sustainable as a human. But no one really was overlooking at the rotation or pages or anything like it. And I know that the same was for a lot of other SREs at that company.  And so many other friends that I know in SRE were like, they completely burnt out because they were constantly getting paged and woken up in the middle of the night that a lot of them ended up extremely depressed, ended up in hospitals, like, I was one of those. And it's one of those things that it sucks because you can't really bounce back from that easily. My story has been really complicated. And for a lot of others, they completely left tech or even just getting triggered by the sound of PagerDuty and things like it. I think we don't consider the big impact that it actually can have on the livelihood of a human. JEN: And I think there is even some PTSD where even when you're not on-call during certain weeks or not on-call anymore when your job changes...like, after I wasn't on-call, I still struggled for...honestly, it was like a couple of years after being on-call for the majority of my life where I'd wake up in the night, and I'd feel panic, like, oh my gosh, did I get a page? Did I miss something? I'd be like, oh, I'm not on call. It is very stressful. ANA: It goes back to the idea, too, that your body's holding in a lot of that trauma that you went through. And as you go through workplaces, you pick up stuff. So it's just the larger whole...like a whole system that's also very complex as you work on the complex systems in technology too. ADRIANA: On a similar vein, was there a time in your career where you were struggling with something like a personal issue where you found that you got some good support from your workplace where you're like, oh my God, thank you, like, I couldn't have done this without a supportive manager, colleague, whatever? JEN: Yeah. So about a year and a half ago, I went through a divorce, and I was at my current role. And I don't know if any of you guys have been through divorce before. I'm sure people listening, you know, there are plenty of people who have. But it's very disruptive to your life. And I told my people, manager, I told my team, you know, "This is what I'm going through right now." And I felt my effectiveness dropping. And my team all supported me. And they were like, you know, there were other people on my team that had been through it.  And I had people on my team saying, "Let me know what you need. How can I help? Do you need me to pick something up for you?" I really appreciate having had the support of my team to get through that because when you're going through something like that, you just can't give that 100% to your job at that time. And I felt bad about it, but I just didn't have it inside me to give. So it was really a relief to be in an environment where I had a group of people that cared about me as a person and as a co-worker and were able to just rally around me and say, "What do you need me to do? How can I help?" ADRIANA: That makes such a huge difference. I can think even last fall; I lost my mom to cancer. And my team was so supportive around that, like, people reaching out to me and just having other people who had been through it offering to talk about it, like, having that support. Like, even though you're going through a really rough time, just knowing that there are others who have gone through it and are there for you, I think, makes such a huge difference. ANA: I think it's that simple I see you, I hear you, I'm here for you. That just validates your human suffering so much. JEN: Yeah. On the flip side, I've also been on the other side. And as an early career engineer, my oldest son had mental health issues. And he was on medication from the time he was, I want to say, almost five, which is pretty unusual, but he was really having some major challenges. And I've also worked in places where I did not feel that I had that support. And it's very difficult. And it impacts your own mental health when you're dealing with somebody else who has mental health issues, and you're trying to maintain a full-time job that's demanding.  And talking about on-call, I was in on-call situations for a lot of that. And I got to a point when my son was 26...I know I'd shared this earlier, but my son did pass away. And I was in a job at the time, but I chose to leave because I didn't feel that I would have that support. And I did take some time off afterwards, which I absolutely needed. At that point, I had kind of made the decision when I go back; I'm going to take the time to find the right role to go to and the right company and the right team. And that's when I ended up where I'm at right now. ADRIANA: That's awesome. And the moral of the story is there are nice companies to work for out there. And so you shouldn't shortchange yourself and stay and work under hellish conditions because there are better things out there. JEN: Yeah, absolutely.  ANA: There is going to be a place that lets you bring your whole self to work and allows for you to show up and request the human needs that you have. And when you realize that that's not a fit or you start having these spidey senses, physical senses of I don't feel so well because of the way that I'm being treated and such, kind of try to trust your gut as much as you can so that you don't end up burning out. JEN: Yeah. If you're not in a place where you can be authentic at work, and you can be vulnerable at work, at least with the people closest to you, to me, I think you probably would be better off in a different position. And it's not always easy to find the right spot. But maybe circling back to, you know, it's okay to do what's right for yourself and take care of yourself. It's okay to look for a spot that you're going to be able to show up and be your authentic self. ANA: And I think it extends to it's a job; put yourself first. At the end of the day, like, what's a job when you don't have you? And I say that speaking to myself [laughs] in the sense of really make sure that you're tending to your own needs, whether it's making sure you're going on trips, taking time off, spending time with your loved ones, having set work hours, making sure that you go to the gym and you do bouldering and things like it. What makes you happy that you can continue being happy to be successful at your job, and does that align with what work allows you to do? JEN: Yep. When you look at where you're at, are you being true to the values? And that is a huge piece of it. And are you being true to your values when you're at a certain company as well? And maybe even doing some self-exploration to try and figure out what are my values and making sure you're in alignment. ANA: It's funny you mentioned that because I think not many people go through exercises of defining their values, like whatsoever. I've done it a few times. And I'm always kind of surprised, like, very different surveys leave the same results, or how I feel about them afterwards. Or keeping in check of, like, am I staying true to what it is that I really wanted to do?  But at the same time, it makes me wonder, is there a space where organizations do these exercises with you as you're coming on, as you're interviewing? So that you, as a candidate early on in the process, are actually able to say, "Oh, holy crap. I'm not even supposed to be doing this." Or "This type of role doesn't allow for me to give back to my community, and that's in my top three values. Hold on, how is it that I can do that in my own volunteer time, or are there other ways around it?" ADRIANA: Yeah, I totally agree. And I think it underscores the point, too, that when you're interviewing for a position, it's not just the company seeing if you're a good fit for them. It's you making sure that they're a good fit for you. Because I think we often get caught up with job descriptions and salaries and sometimes desperation to leave where you're at, and then you end up going into something equally bad or worse. So being aware of yourself, what your values are, I think, makes a huge difference. JEN: I've gotten caught in that in the past where I got drawn in by money and title. And I thought, you know, I'm not sure this is the exact company for me. But I go there, get the money, get the title, work there for a few years. When I did that, and, you know, I'm thinking of one thing in particular, I found that that was definitely a big mistake. And there are many things that are more important than getting that next jump. You can get that next jump somewhere else without sacrificing, you know, taking care of yourself and what's important to you. ADRIANA: Yeah, totally. And taking care of yourself...my PSA for the day is take your damn vacation days.  ANA: Ooh. ADRIANA: So many people I've talked to...and I worked in consulting for five years. And the company I was at there was mega hustle culture. So there was like so many work...there was like, oh, I didn't take my vacation at all this year because I was doing it all for the project. And it's like, um, yeah, I'm going to take all my vacation days, thanks.  ANA: [laughs] ADRIANA: I value my time off. [laughs] It's my body saying, yo, it's time to chill. It's time to relax. ANA: Same. I very much think you need to relax in order to do your best work. And I am very big on being outspoken about me taking time off and telling other people around me to take time off. So I'm known to tell my managers like, "You haven't taken time off in a while. ADRIANA: [laughs] ANA: Like, what are you doing?" That's always gone very interesting because it's just not the standard, or also; it's a hot take. But for me, I go up to leadership like HR, C-levels, and I'm like, "Are you guys tracking the unlimited PTO you guys are giving us in the sense for the goodness of the employees as in making sure that they're actually taking time off?" For them to actually be like, "Hey, Adriana, you've worked about seven months. We see you've taken two days off. How about you go take a mandatory week and a half off? Thank you. See you in a few weeks." That's what we should be doing. JEN: Yeah, even on a small scale, I think it's great. If you can find times to tell someone out of the blue...and not even like, "Hey, why don't you take PTO?" But just say, "Hey, go home. Get out of here. You got a long week. Get out of here. I don't want to see you again until Monday."  ANA: [laughs] I always like those on Thursdays where it's, like, just take Friday off. And it's like, oh, it's like school just got canceled randomly.  ADRIANA: I know, right? ANA: And it's a sunny day.  [laughter] JEN: Of course, you wouldn't have snow days but --  ADRIANA: Oh yeah. One thing I'm hoping for is to normalize the four-day workweek because I know a lot of companies are starting to adopt it, not as many as I would like, but definitely, the numbers are growing. And I think was it in Iceland that they ran a national experiment of the four-day workweek? And it was super successful. So I have high hopes maybe it'll make its way into North America as the standard. ANA: I also think so. I mean, I've seen some technology companies moving to four-day workweeks. I would also like to see it or at least be given the option to partake in it.  ADRIANA: Yes.  ANA: Because I personally feel like my mental health could actually really appreciate just longer work days with more focused time but having that extra day and having two rest days and one reset day to prepare for the week. And I think this conversation also matters more with the COVID era that we're in where it's just like, our lives are so much more different. And because most of us still end up spending a lot more time at home than we did four or five years ago, I think more than ever, you need an extra day to either clean up your house a lot more or get the hell out of your house or just change up something. ADRIANA: Yeah, hear, hear.  ANA: [laughs]  ADRIANA: Oh, we're coming up on time. But there's one more thing that I wanted to touch upon with you, Jen. When we did the pre-chat, I think you mentioned that you work with a mentorship organization. JEN: Oh yeah. I've been for about a year and a half part of...it's a nonprofit organization. We're called Cloud Girls, and we advocate for women in the tech space. So each year, we open up for nominees for awards, and we grant three different nominees for women in different areas of their career growth based on something great that they've done that year. We also, each year, we'll pick a different organization that we can sponsor, and we can do fundraisers for. It's been about a year and a half that I've been a member of Cloud Girls. ADRIANA: That's awesome. I definitely want to check it out. ANA: That sounds really cool. And I always feel like names like that come with really cool stickers. Like, I did a lot of those programs growing up too. But bringing in more underrepresented folks into this space and actually providing mentorship for them because it's not just about opening the door for them, it's about actually setting them up to be successful and have someone to talk to about their experience, the ups, and downs of it. JEN: Yep. And it is hard being a woman or in a different, you know, underrepresented group in tech because you don't have as many women in senior positions. So it's a lot harder to find that mentor. So in response to, I guess, my own comment, it really shows the real value too of having allies, male allies, to help mentor women and amplify their voices as well. ADRIANA: Yeah, absolutely. And hey, who knows? Someday it'll be normalized to have a bigger representation of ladies in tech. Wouldn't that be nice? JEN: Yeah, I think it's especially a challenge in the SRE DevOps space as well. Because when I look at some of the other capabilities that I work with, software engineering, and data engineering, and experience design, there are more women just in general, even though it's still underrepresented, but there are more women in general in those fields. But for whatever reason, it seems like that SRE DevOps space, we still struggle more than some of these other areas to get women. ANA: It's like we go from being one of the only ones in the room to being one of the only in the auditorium. Like, we just get smaller and smaller that when we see another one of us, it's just like, oh my God, hi. Please say hi. [laughs] Like, can we be friends? Are you suffering too? Like, are you okay? [laughter] Can you give me a hug? ADRIANA: I know I get so excited when I meet other ladies in tech. It's like, yay, there are more of us. [laughs] ANA: My favorite is running into [inaudible 37:58] girls, SREs, or even non-binary folks in the bathroom where it's just like, it's empty. You are here. You're cool. We are friends. [laughter] JEN: Yeah, like a tech conference is the exact opposite of the bathrooms like at a concert, you know? ADRIANA: Yeah, exactly, exactly. Yeah, that's probably the best place to meet [laughter] other badass ladies in tech, the tech conference bathroom. [laughs] ANA: I think there's only been one technology conference that I've been to that I had to do a line for the bathroom, and I want to say it was actually one of the...I think it was the second day of KubeCon North America. And while I was in line, I remember telling the other folks on line like, "Oh my God, I'm doing a line. I'm so happy." [laughter] I've never been happy about doing a line. [laughter] ADRIANA: Well, I think that brings us to the end of our conversation with Jen. Thank you so much for joining us in today's podcast. Folks out there, don't forget to subscribe and give us a shout-out on Twitter and Instagram, and Mastodon via oncallmemaybe. Be sure to check out the show notes on oncallmemaybe.com for additional resources and to connect with us and our guests on social media. For On-Call Me Maybe, we're your hosts Adriana Villela and... ANA: Ana Margarita Medina. Signing off with... JEN: Peace, love, and code. ANA: Woo! ADRIANA: Yay. [laughter]

About the guest: Michael Kehoe is an author, speaker and Sr Staff Security Engineer at Confluent. Previously he was a Sr Staff Site Reliability Engineer (SRE) at LinkedIn architecting LinkedIn’s move to Microsoft Azure. Before graduating with a Bachelor of Electrical Engineering from the University of Queensland (Australia), Michael interned at NASA Ames Research Center building small-satellites known as Phonesats. While working at LinkedIn, Michael led the company's work on Incident Response, Disaster Recovery, Visibility Engineering & Reliability Principles. He has also been embedded with the profile, traffic, espresso (KV Store) teams. After leading LinkedIn’s last physical data-center build, he is now the architect for how LinkedIn builds its infrastructure in Azure. Michael has spoken at numerous events all over the world and has authored the books "Cloud Native Infrastructure with Azure" and “Reducing MTTD for High Severity Incidents”. Find our guest on: Twitter LinkedIn GitHub Personal Blog Find us on: On-Call Me Maybe Podcast Twitter On-Call Me Maybe Podcast LinkedIn Page On-Call Me Maybe Podcast Mastodon On-Call Me Maybe Podcast Instagram On-Call Me Maybe TikTok On Call Me Maybe Podcast YouTube Channel Adriana’s Twitter Adriana’s Mastodon Adriana’s LinkedIn Adriana’s Instagram Ana’s Twitter Ana's Mastodon Ana’s LinkedIn Ana's Instagram Show Links: Confluent LinkedIn SRE Lightweight Directory Access Protocol (LDAP) eBPF Liz Rice General Data Protection Regulation (GDPR) Terraform tfsec (security scanner for your Terraform code) Cloud Native Computing Foundation (CNCF) Azure Policy Open Policy Agent (OPA) Kubernetes Admission Controller Additional Links: Cloud Native Infrastructure with Azure Reducing MTTD for High Severity Incidents   Transcript: ADRIANA: Hey, y'all. Welcome to On-Call Me Maybe, the podcast about DevOps, SRE, observability principles, on-call, and everything in between. I am your host, Adriana Villela, with my awesome co-host... ANA: Ana Margarita Medina. ADRIANA: And today we are talking to Michael Kehoe, who works at...I don't know where you work. [laughs]  MICHAEL: Confluent. [laughter] ADRIANA: Confluent. Excellent. Welcome, Michael. [laughs] MICHAEL: Thank you, ladies. It's so great to be recording with you both. ADRIANA: So, first things first, what are you drinking today? MICHAEL: Today, it is just water. I just got over a cold recently. So we're recording this in the middle of the day. So water for now, but I've got some coffee liqueur to finish before I get on my plane tomorrow. So I think that will be my after-work drink this evening. ADRIANA: There you go, goals for the end of the day.  MICHAEL: Absolutely. ANA: That seems really tasty to look forward to towards the end of the day, so I'm definitely a bit jealous. For me, I'm actually very much feeling in the very festive mood. And I decided to get a white chocolate peppermint mocha with peppermint from Starbucks.  ADRIANA: Yay.  ANA: I'm on that holiday cheer kick today. ADRIANA: Hooray, hooray. I've got a can of Perry lime with me and some water to supplement. So not super exciting, I'm afraid.  ANA: [laughs] ADRIANA: But hydration is hydration, so good vibes all around. [laughs] All right, cool. Well, Michael, we always like to hear from our guests, like, how'd you get into your current career path? MICHAEL: When I was very young, four or five, I had an interest in computers. And during school, I had the opportunity to tap into that a little bit, but not too much. When I was in college, I got an opportunity to work for my university's IT department. I went to school back in Australia, and the universities are generally much larger there, so we're talking about a user base of 40,000 students, 10,000 staff across dozens of campuses.  This role was in their network department. I started doing low-level tasks, making sure that switch ports worked in offices, setting up network switches. And that got to evolve into helping build out new data centers. It also allowed me to go and do more network engineering tasks. So my university was an ISP as well for not only the university but also commercial customers as well in the form of both residences but also other universities, other schools.  So this gave me a lot of experience to go and help build out actual solutions, build customer experience, customer service mentality a little bit. I got some exposure working with our Linux team as well. So I got to learn Puppet. I got to learn a little bit of LDAP and was able to start putting these different skill sets together.  At the same time, Google recruiters came to campus, and they had a presentation on SRE. And I'm like, oh, this is so cool. You get to do a little bit of coding, a little bit of Linux, a little bit of network, and problem-solving as well. And I really loved the combination of those skill sets. This is back in about 2012, 2013. There was this blog, I think it's still around, called "High Scalability," which talked about real-world architectures of different tech companies.  And I immersed myself in that to learn how all these things work. And so towards the end of college, I did some job interviews, and, thankfully, was able to interview at LinkedIn for an SRE position, and thankfully got the job and then went from there. I was on the Profile team at LinkedIn, and then from there went on to a more central team that handled more infrastructure across the whole site and practices and procedures. And during that time, I also got seconded to a number of different teams to help them through some trying situations.  So I got exposed to everything from the Ingress traffic layer all the way to backend key-value databases and grabbed a bunch of experience across teams, which was really awesome to be able to learn such specific knowledge across a variety of different experiences. ADRIANA: Cool. So you actually got an SRE role right out of college, then.  MICHAEL: Yes. It was really challenging to do that. A lot of companies won't hire SREs out of college because it's very difficult to get that experience in college unless you've done an internship where the bar is a little bit different. But yes, I graduated and got on the first plane after grabbing my visa and got to work. And I'm very grateful for the opportunities I had at LinkedIn, especially to be able to just immerse myself in so many different areas of the stack. ADRIANA: That is wild. ANA: You also got a chance to come into SRE at a prime time. It hadn't necessarily kind of picked into a lot of big companies starting to use it, and the book had not come out yet. So folks were kind of still like, oh, there's this thing that you do to keep systems up for humans and Google calls it SRE. And, I mean, at that time, knew that Facebook called it production engineering, but that was it. Like, nobody was really trying to name-grab or anything. It was a really interesting time, so... ADRIANA: Probably wasn't solid like it is now [laughs] because it was kind of at the forefront, right? MICHAEL: I mean, I think at the time, there was a loose definition of what it was. And for those hyper-scale companies like Facebook, like Google, like LinkedIn, and others, I'm sure they worked out what made sense for their companies. At the time, I think a number of other companies were like, what is this thing? And post-Google SRE book publication, companies are now trying to work out how do I make this work in my organization?  And we definitely seem to be at the point where like, okay, this book isn't canon; it is meant to be a guide. And each company finds their own way to put these various principles into practice in a way that makes sense for both the culture of the business but also in terms of how the business is run. ANA: 100%. And I think that we're still seeing that shift to people realizing that it's not a one-size-fits-all. I think we've had a lot of those conversations in our podcast recently where it's like, nope, you can't just grab the book and apply all to all your systems. It's all a lot of hard work. And as you said, culture comes into play. But the way that Google does it, the way that Facebook or LinkedIn does it does not apply to your enterprise company or your startup. MICHAEL: Right. A friend of mine who worked at LinkedIn also has his own podcast series. He has a quote that "Culture eats technology for breakfast," which is very true. ANA: I know that you, like, I met you when you were working at LinkedIn. You were working specifically on EBF and just sharing SRE practices. What do you feel about that space now, considering EBF observability has taken a huge rise? And as you were talking about it in 2018, you were the only person at conferences speaking on it in a sense. MICHAEL: Right. I wish I had more time to spend on it, [laughs] honestly. So eBPF, when I was talking about it at, I think, O'Reilly Conferences and also the SREcon series it, was in its infancy in the industry. The foundations of eBPF started in; I think, 1991. And then the first commits to what's now known as eBPF was late 2013 or early 2014. And so it took a couple of years for people to start picking it up. And now you see companies specifically like Cilium that have really accelerated its growth, and they've created a very well-known product.  I think the great thing about that space is it is more or less an open-source technology eBPF. So anyone can go and create something exceptionally powerful with it, which is great. Obviously, Cilium has cornered the market and makes it more user-friendly to go and use in a Kubernetes containerized environment. But there's nothing really stopping me from going and building a high-performance load balancer or DDoS system if I really want to. I think for the space that I'm in now in security, we can go and do very low-level, kernel-level auditing of system call events, or go and do deep introspection of network flows without having to worry about the overhead of that software. And that's so exceptionally cool. So while Cilium has cornered the market for now, I can't see a reason why other companies won't come out with eBPF-based network flow monitoring and different IDS solutions that fit a broader market than just platform-hosted Kubernetes. ADRIANA: Now, for our listeners who might not have heard of eBPF or have heard of it but aren't really sure what it is, can you provide a super high-level definition description? MICHAEL: Sure. So BPF essentially allows you to run your own programs in kernel space without needing a kernel module. The best example of this is tcpdump. You define your own filter, and that filter actually gets compiled into kernel bytecode. And then the kernel actually goes and pauses the packets coming through your network interface, rather than going and doing that in user space, and that is massively efficient.  So that concept is not particularly new. tcpdump has been around since the early '90s. But with eBPF, you can go and create your own filters and programs, go and create those in userspace. And then, they go and get compiled and get run safely in kernel space, which generates the efficiency of the program. So you can go and do network-level parsing, load balancing, or even look at all the system call events happening on a machine, do that in kernel space, get the results back in userspace, but do it at exceptionally high throughput, which makes it very attractive to system operators such as myself. ANA: That's super awesome. I mean, it gives a lot of flexibility, and you get to use it in what ways works for your team. MICHAEL: Absolutely. There are endless ways you can take it. Definitely, the bar to entry has been a little bit challenging, just because you need to be able to write programs that actually run in the kernel, so understanding all the intricacies of those C or Assembly if you're really eager for these programs. It is a little bit more challenging, especially for... I've written some network programs, you know, you need to understand the packet structure exactly to make things work. But yeah, if you know what you're doing, the possibilities are endless. And I think that's what makes it really exciting. ANA: I haven't been following too much of the trends around it. But are we also then seeing, I guess, this being used a lot more in the security space? MICHAEL: Definitely for doing container-based networking, yes. And Cilium also allows you to do the identity-based policies as well, which I think is very exciting. Because you sort of say, all right, this service can talk to this service, and a bunch of layers of abstraction take care of that for you. I think there's still a space where...the hyperscalers have definitely done it. Like Cloudflare, if you go and look at the eBPF tag on the Cloudflare blog, you'll find endless amounts of ways that they've used eBPF, probably the same with Facebook as well. The rest of the industry, we really haven't seen eBPF IDSs come to the forefront yet where we can really do deep introspection.  Falco uses eBPF as well to do some of its system monitoring. So we're getting there, but I don't think we're going to fully mature for probably another two or three years. I know Liz Rice, who is another person who has written on eBPF before and also works at Cilium; she's working on an O'Reilly book at the moment, which is really going to take us back to first principles and walk people through how to build their own programs. So I'm sure when the publication of that book happens next year, we'll again see a jump in interest in the space. ANA: That's exciting. One of the questions we also haven't touched upon is how did you transition from SRE to security? MICHAEL: While I was at LinkedIn, I had an interest in the space. LinkedIn, I think they still do it, had a Security Champions Program. So they took someone from each engineering department and allowed them to go through a quarterly training program. So you did some online Stanford cybersecurity courses and got paired with a security team member to go and work on a security-based project in your engineering team. So I did one of those programs while I was at LinkedIn. While I was working on some of my projects towards the later part of my tenure at LinkedIn, I was heavily engaged with the security team on helping make our cloud infrastructure secure there. When I was looking for a new role, I was really excited to go and do something a little bit different, something a little bit new, and somewhere where I had a lot of space to grow and a lot of things to learn. So making the jump to Confluent and joining their security team was an exciting step. I definitely have learned a lot to date, especially since Confluent is a true multi-cloud vendor. So there are a lot of interesting challenges to be solved there. ADRIANA: To continue on the security thread, I feel like, in some ways, security is a thankless job, right? Because I've been in companies where security policies are enacted, and developers are like, "What the hell, man? I need to be able to do my job for blah blah blah blah blah." But then there's like all this stuff happening behind the scenes that security folks can't necessarily talk about. What are your thoughts around this? MICHAEL: So there's definitely a balance for infosecurity professionals defined in their roles. For myself, I see my partnership working on two levels, sort of similar to SRE, number one, the business needs our role to be successful for the business to keep growing. Working for a SaaS vendor, customers want certain requirements of our platform, certain guarantees, and it's up to us to help ensure that those are in place.  For the internal customers, which is mostly engineers, we definitely see our role as building a partnership. Yes, we need to put various controls and constraints in place. But how can we go and do that in a developer-friendly manner? For some of the things we have enacted, and I won't go into specifics, we've definitely consulted with various engineering partners of, like, okay, we need to do this. How can we make this user experience work well?  And so definitely taking that SRE mindset of how can we make the business successful on multiple fronts and bringing that to security has definitely been a focus of mine. We don't want to be the organization of saying, "No," we want to be the organization of like, let's work together and find a solution that suits the business across multiple different departments. ADRIANA: Cool. And have you ever encountered a situation where you're like, you know, I want to enact this security policy, and then you start talking to developers, and they're like, "No, no, no, you haven't thought of XYZ." And how do you ensure that you keep things secure while still keeping the developers happy? MICHAEL: That's a great question. So the thing that first comes to mind is a quote from Star Wars that "Only siths believe in absolutes," and security is sort of like that. Like, there are very few absolutes in anything that we do. Again, there are definitely policies that we always want to put in place. And it is challenging sometimes to be able to go and say, "I'm going to blanket-apply this policy."  For any project that we're doing, we're now by default putting a quarter aside of work just to do research on like, okay, we want to go in this direction. What impact will it have across the business? And so by the time it comes to going and executing on that, we're not going to upset a bunch of people. Like, we've already planned and accounted for it. ADRIANA: Oh, nice. That's refreshing. MICHAEL: Right. Changing our stance to being more expecting problems rather than going directly to escalations when we're trying to execute that has definitely been helpful. And Confluent is very good in its planning process. So before the quarter happens, we can go and talk with other teams saying, "All right, this is the direction we want to go in. For the next quarter, we're going to work with you to work out how we can make this policy happen. And then the quarter after, we'll need your partnership to go and execute on it." So that model has been very helpful for us in reducing the number of conflicts or potential problems that we have in the execution of our projects.  That being said, also just being a great partner in being involved in the design of projects as early on as we possibly can definitely help. Again, we're a vendor in a multi-cloud space that comes with a number of just unique challenges on the design of our infrastructure. So being involved in how some of our very low-level infrastructure works and then you design for those at the beginning definitely ensures that we can be much more successful later on. And instead of things having to go through a design review just before the projects are meant to go live, that happens in the months before. And we avoid surprises at the end. ADRIANA: The true shift-left as DevOps intended. It's so refreshing to hear that. That's awesome. MICHAEL: Right. We've taken that shift-left approach as well to our on-call. On-call is very different to SRE on-call, I found, which surprised me. I guess I didn't think about it a lot. Definitely, for any alerts we receive as a security team, we're looking at ways of, like, how can we automate this away? How can we make this more friendly to ourselves so that we don't have to get woken up by something in the middle of the night?  In the SRE space, I can kind of limit what changes are made to the infrastructure and keep that until the business day. But security people around the world or researchers or adversaries they don't really care about my nine-to-five. So we have to come up with ways to make our on-call exceptionally efficient so that we don't get burnt out during our on-call stints. ANA: How else have you felt that is kind of different in terms of the principles that you advocate for working in the reliability space versus a security space? MICHAEL: I think for reliability, it's a little bit easier to justify what you're trying to achieve because the site does need to be up, or the infrastructure needs to be up for you to make money. Security is a little bit more difficult to justify why you're doing something and its impact on the business. For confluent, we need our product to be secure for our customers to want it. And I think the company as a whole does understand that, which makes things easier.  But, again, making all these things happen really means that we have to be a very plugged-in partner for other teams. Since I joined, we've grown our security team very rapidly over the last 18 months. We've tried to be very diligent in that planning process, in that partnership process. Everyone now understands that, yes, security is very important to the growth of the business and also, of course, to our customers.  And so, now, instead of coming to people when there's a problem, we can tell them in advance, "This is the roadmap that we want to go down over the next time period. This is where we're going to need your partnership, and this is what we're going to be asking of you." And that has been a very successful way to ensure that we have buy-in from all those teams and so that they can also plan for some of the restrictions, or policies, or controls that we want to put in place over a period of time to uplevel what we do on a daily basis. ADRIANA: Now, do you typically have security folks embedded with development teams as part of the work, or is it more of a team that services multiple teams? MICHAEL: That's a great question. For our application security team, they're definitely very much embedded, not in a formal sense, but they're very much on the pulse of what different teams are doing. Confluent also has an on-prem offering as well. They need to be very plugged into release cycles, bug management, vulnerability management, et cetera.  For my team, which is the cloud security team, definitely, we've made a bunch of partnerships across the company. So people know to come to us if they've got a question. Over the period of time, we want to get more involved and formalize some of those partnerships. But we're still a very young security team, and we're still growing. So hopefully, in 2023, we can formalize some of those partnerships. And it would be great to have a similar model to what we did at LinkedIn and have those cloud security champions or application security champions in different teams that can be our eyes and ears on the ground on a daily basis. ANA: That'd be cool to be able to see you completely start up one of those champion programs because it's true, that's one way to help make a practice a lot more sticky in an organization but to also uplevel folks and be like, this is something you're interested in, and you might want a career in it. MICHAEL: Right. We actually have a cross-team collaborator award that my organization gives out every month. So we give, I think it's a gift card to someone who's really helped us over the last month. So we definitely would like to recognize those people in the company that are working to help us meet our goals. And I'm sure at some point; we'll expand that into formalizing some sort of program and really solidifying those relationships and also maybe doing some recruiting on the down-low. ANA: With your role currently, do you find yourself also having to build software, or do you see it more as enforcing policies, reviewing? MICHAEL: Great question. I haven't done a whole lot of software development while I'm here. That being said, I am rushing to finish one of my software development OKRs before the end of this week. My team definitely wants to build more software to help make our lives run smoother. Building software is really important for my team so that we can help enforce policies. So we need to do a bunch of compliance tasks. Being a vendor doing those tasks manually is not necessarily fun.  So we're actually building software to go and ensure that, yes, we're ticking these boxes. Yes, this thing has been done. Yes, we're monitoring these things automatically instead of manually. So that has definitely been a focus in our recruiting when we hire people. And then OKRs of, all right, how can I automate this away? How can I make this thing show up by default so that our jobs and the work that we do is scalable as the business grows? AWS, GCP Azure, they keep opening new regions. Our footprint in those regions keep going up quite rapidly. The way that we secure the infrastructure also needs to be scalable with that infrastructure growth. So we're trying to build preventative and monitoring measures for different parts of the business and different parts of the engineering process that scale in a very hands-off manner. ANA: With your work working on many different cloud providers and Confluent being a multi-cloud vendor, what are some things that companies can be doing currently to stay more secure? MICHAEL: I think it comes down to doing fundamentals very well. Thinking about if I'm designing a system, how is the system going to scale over time? What are the points of manipulation in the system, and how can we, number one, audit what happens in that system for both regular actions against a system but also malicious actions against that system? And then looking at how can you put preventative controls in that system? If people did that when building things, that would make my job sort of obsolete mostly. A lot of what my team does is work out how can we ensure that the system doesn't get abused? And if it potentially does or if we need to do an investigation, how can we ensure that we have the data to go and do that and draw positive conclusions? A lot of how we should be building infrastructure in going forward needs to be thinking about putting different access controls in place, like our backend systems.  Now, a thing in the industry, especially with the rise of GDPR and also different policies across different U.S. states, is thinking about access control, auditing from the beginning, and building that into the product or the system that you're building really makes a difference in ensuring that it is scalable over time in the security space. ANA: It makes perfect sense. Is there any other metric, or observability, or monitoring that kind of also should be getting added to these procedures or these processes or systems? MICHAEL: That's a great question. We're actually starting to build security scorecards for different teams. So we have a number of different measures of how well the team is doing on various security goals that we want to see. I won't detail all of it publicly. But that gives engineering teams, engineering leaders a good insight to see where we want to see those teams get to over a period of time or how quickly some of those things get implemented as well because that does matter to us. We've definitely seen in the SRE space...like, at LinkedIn, we had this thing called service scorecard which was like a list of 50 different requirements that we'd like to see your service have. And that was all from dependency management to load balancing preferences, you know, having ownership data correctly listed, et cetera. So we're starting to build something similar. And I've definitely seen a few companies pop up offering similar things, mostly in the productivity engineering space. But I wouldn't be surprised to see it come to security at some point in time, a startup idea maybe for someone out there.  ANA: [laughs] I've also seen the productivity ones. And I definitely agree, like, on security, they should be forefront, or even just seeing a little bit more of tying it all together from security reliability, and developer experience, everything that goes behind the scenes of operating systems. MICHAEL: Right. As we were talking about earlier, SRE is very much a role where you're serving different parts of the business to make it successful. Security, honestly, isn't that much different. It's a slightly different space. But security and infrastructure reliability are more intertwined than people probably give on first thought.  And I've definitely been in situations in my current role where I've been thinking about how do I make this infrastructure reliable but also secure at the same time? And thankfully, in some of those design meetings, I have the SRE experience to draw on and help those engineers make good design decisions from early stages of implementation. ANA: It's funny you say that because that was actually one of the questions that we planned on asking you. Like, as you have done SRE and security work, what is that intersection? MICHAEL: Closer than you think. My team definitely is very involved in the running of different security pieces of our infrastructure. We don't necessarily run all of those ourselves, but other teams do. So we're very invested in ensuring that those systems are reliable and also auditable. We definitely have these partner teams where we have partner infrastructure teams where we're very invested in what their roadmap looks like and how we can ensure that what they're doing to scale our infrastructure can be made more secure over a period of time. ANA: With you being able to work on SRE and security in the past, what are your thoughts around the DevSecOps movement? Is this different from what y'all are doing right now? MICHAEL: I think this is a really untapped area. So definitely, the DevOps space got all on board with Terraform. I have a lot of personal thoughts about Terraform. ANA: I mean, podcasts are the perfect place to put all the spicy nuggets. I'm just going to throw that out there. [laughs] MICHAEL: I'm more than happy to talk about this. I've definitely had people come and have conversations with me and use the word Terraform like it's a Jedi mind trick, [laughter] and so it's like, Terraform fixes all your problems. And Terraform definitely has its place, but Terraform by itself does not do that. Allowing people to go and write Terraform code and then run Terraform apply from your desktop is not that much more secure than allowing them to do that from a CLI or a portal. All it does is make someone maybe commit it somewhere so people can work out what the hell you're trying to do. And so you definitely need systems that audit that behavior, which Terraform Enterprise provides that. That is also ungodly expensive. For the security space, we definitely see infrastructure as code as a good place for us to be very proactive about our stances.  There is now tfsec and a couple of other different projects out there where we're able to specify policy in pre-commit and tell people, "Okay, you shouldn't do that." So a very classic example is the creation of S3 buckets. You should not be able to create an S3 bucket that is public to the world. And instead of having a monitoring control on that or some sort of latter control that happens after the deployment of that S3 bucket, we can say at pre-commit time, "Hey, developer, this is not a good idea. You can expose S3 data to the world, but you have to do it in this way with these approvals." ADRIANA: So you're putting some guardrails in place. MICHAEL: Yeah, absolutely. The cloud providers are getting better at this as well. Like, I really love Azure Policy, where you can specify very granular policy across your cloud infrastructure and also script that to different management groups or subscriptions. That is a great preventative control. AWS is getting there.  But bringing the experience to the developer at pre-commit time, I think, is a really good experience and especially because those tools can allow us to point to that internal documentation of what to do instead of getting a blanket you're denied from doing this. So I think the DevSecOps space can really leverage IAC in a very positive manner.  And then there are a bunch of other security projects that are part of Cloud Native Computing Foundation that also enable other security features like signing images, et cetera, and also admission controllers as well. They're very important parts of the security worldview that have now become pretty baked into the development experience and the developer workflow. ANA: That's really cool. I do like also what you mentioned of being able to tie it into your internal tooling with that pre-commit because you definitely just kind of want to guide them into, like, hey, it's not like a hit to the hand. It's just more of a let me guide you to the best practice that the organization actually wants you to do. MICHAEL: So even this scorecard product we had at LinkedIn only told developers that they weren't following best practice after they had developed or deployed something. With the DevSecOps movement, moving that all to pre-commit goes a long way to ensuring that, number one, engineers are provided with actionable feedback and that it also becomes a blocking mechanism for them to get their pre-commit or get their commit pushed.  So this becomes a really good forcing function where we can helpfully tell the developer or engineer what to do and ensure that that gets done. So the best practice is adhered straightaway rather than something being deployed, we find it, then we have to go and ask them to go and fix it, which becomes a pretty bad cycle, especially at scale. So there are a number of different and interesting ways that we can be much more proactive and ensure that everything is secure by default rather than having to chase a team to go and adhere to a best practice retrospectively. ADRIANA: Which is quite refreshing. Before we wrap up, I did want to ask you a question about policy as a service. Is that something...because you mentioned Azure has some mechanism to let you define -- MICHAEL: Yeah, Azure Policy.  ADRIANA: Yeah. And that AWS is kind of getting there. Are you aware of any tools out there that are kind of cloud-agnostic that give you that policy as a service thing? MICHAEL: Definitely, there's an Open Policy Agent, which is a CNCF project and can be plugged into a number of different things. I think it can be used as an admission controller for Kubernetes. It can be used as a policy evaluator for SSH access. So there are definitely projects out there. And OPA is just a REST API essentially.  So I think when these systems become a little bit more mature, especially in the way that policy is crafted, OPA I found was a little bit trickier than I expected. Once these projects get a little bit more exposure and we can plug them into a whole variety of systems very easily, then we make life much more simple. And we don't have to worry about applying policies retroactively.  ADRIANA: Cool.  MICHAEL: We also centralize the policy, which is also very helpful. ADRIANA: Yeah, yeah. That's cool. So I guess keep an eye out for OPA and other such CNCF projects out there. It's only going to get better from here.  MICHAEL: Right. ANA: It's always cool to see how many other projects there are within the CNCF umbrella. Every time we record, I feel like I learn about a new one or a second one. And I'm like, whoa, [laughs] this space keeps growing.  ADRIANA: I know. It's wild. MICHAEL: Right. There are obviously a lot of projects in the CNCF space. There are definitely a number that have a lot of potential to really make a huge difference, especially to smaller companies to do things that the hyperscalers were doing a couple of years ago. And they can do that from day one. One of the benefits that startups have is they are completely greenfield and are able to be a lot more nimble. So maybe one day that will be me, not yet. But I would love that opportunity at some point in my career to build from greenfield with these mature cloud-native projects out of the books. ADRIANA: Yeah, that will be awesome. ANA: Most definitely. Definitely agree there, too. ADRIANA: Cool. As we wrap up, do you have any words of wisdom that you want to share with our listeners on security nuggets? MICHAEL: I think, going back to what I said earlier, only a Sith believes in absolutes, but there are very few absolutes in the work that we do. Security is meant to serve multiple parts of the business. And so the role of a security team is not to say, "No," it's to work out solutions to help the business be successful and meet its various contractual obligations, and take that SRE mindset and try and build off what you know SRE has done over the last 15 years and move things to the left, build those partnerships, use software to make our lives easier and to automate as much as we possibly can. ADRIANA: Love it. Those are really great pieces of advice. ANA: Well, with that, thank you so much, Michael, for joining us in today's podcast.  MICHAEL: Thank you. ANA: Don't forget to subscribe and give us a shout-out on all social medias via oncallmemaybe. And be sure to check out the show notes on oncallmemaybe.com for additional resources and to connect with us and our guests on social media. For On-Call Me Maybe, we're your hosts, Ana Margarita Medina...  ADRIANA: And Adriana Villela. Signing off with... MICHAEL: Peace, love, and code. ADRIANA: Woo-hoo! Yay.

About the guest: Shingi Kanhukamwe is an Executive Transformation Advisor working with Export Development Canada. He has 11 years of Agile experience and 9 years focused on leading transformation initiatives in large, complex organizations. He has worked at various financial organizations and has also worked as a consultant.  Find our guest on: LinkedIn Find us on: On-Call Me Maybe Podcast Twitter On-Call Me Maybe Podcast LinkedIn Page On-Call Me Maybe Podcast Mastodon On-Call Me Maybe Podcast Instagram On-Call Me Maybe TikTok On Call Me Maybe Podcast YouTube Channel Adriana’s Twitter Adriana’s Mastodon Adriana’s LinkedIn Adriana’s Instagram Ana’s Twitter Ana's Mastodon Ana’s LinkedIn Ana's Instagram Show Links: Content Management IBM Digitization Cloud-Native Transformation Sociotechnical systems Mainframe Netflix’s culture explained Transcript: ADRIANA: Hey, y'all. Welcome to On-Call Me Maybe, the podcast about DevOps, SRE, observability principles, on-call, and everything in between. I am your host, Adriana Villela, with my awesome co-host... ANA: Ana Margarita Medina. ADRIANA: And today, we are talking to Shingi Kanhukamwe, who is an independent consultant working on organizational transformation. So, Shingi, welcome. SHINGI: Thank you. I'm so happy to be here. ADRIANA: First things first, important question. We start off for all of our guests; what are you drinking today? SHINGI: I am drinking a kombucha, ginger-lemon kombucha. ADRIANA: Ooh, awesome. Exotic, I like it. How about you, Ana? ANA: I'm honestly always a huge fan of anything ginger. That's actually what I had for my breakfast caffeine which was like yerba mate with ginger tea. It was delicious. But for today's podcast recording, I'm on just regular H2O. ADRIANA: I, too, am drinking water. I was for a previous recording drinking some green tea, but I ran out of that, so water it is. SHINGI: Oldie but a goodie. [laughs] ADRIANA: Exactly, can't go wrong with water. One question that we also like to ask our guests is, how did you get your start in tech? SHINGI: Actually, growing up, I'd always been really interested in technology, so probably starting about the time I was 9 or 10, I got really interested in computers. And I spent so much time on my computer that my parents were getting worried about my future because [laughter] they're like, "You're spending too much time on this thing. Like, is there a future with these things?" To the point where they were just like, "Okay, we're not buying another computer because you just spend way too much time with this thing."  So what that made me do was it made me try and squeeze as much performance out of the machine as possible. [laughter] So I would do all these things to kind of try and get the machine to run programs. Like, I'd boot the machine into DOS and load programs directly from there. But I also got more interested in kind of...I was kind of like, well, what can I do? Well, I can't change the hardware. ANA: [laughs] SHINGI: But what can I do from a software perspective? Like, what optimizations can I leverage to get more out of this computer? Because they're not buying me a new one.  ADRIANA: [laughs] SHINGI: And I actually got interested in terms of, like, oh, I can't buy a whole new computer, but can I buy certain parts? Like, if I borrow more RAM, like, is that going to help?  ADRIANA: Ooh. SHINGI: That's how I got interested in technology from a personal perspective. Weirdly enough, that didn't end up resulting in me studying computer science. But I did a lot of self-study from a networking perspective. So at one point, I was really interested in networking and just thinking about pursuing a career path. So yeah, the first job I got was actually a communications job working on Parliament Hill for a Member of Parliament.  But a big part of the job was actually technology because I ran the website. And back in 2004, there was no content management, [laughter] well, not that there wasn't content management in the marketplace. There wasn't content management where I worked; let's be specific. ADRIANA: Right. Right. SHINGI: So it was this PHP monstrosity where -- [crosstalk 3:24] ANA: [laughs] ADRIANA: Oh wow. SHINGI: You were liable to break stuff when you were just updating the content. That was super fun. So I think, like, yeah, in terms of my relationship to tech career-wise, it's always been like a theme. But I never actually ended up working as a developer or an engineer or somebody who's like in networking in hardware. It was always doing jobs that involved some degree of competency/technical literacy.  So after that, my focus was kind of in the area of digital marketing more from the marketing technology side of things because, at that point in time, some exciting things were happening around marketing automation and what you can do with a modern content management system and analytics, and being able to do things based on triggers.  After that, I started moving more into the organizational transformation space because I had the opportunity to be part of a project where we needed to move away from a very antiquated point-of-sale system with the organization I was working with at the time. So it was an insurance company. It sold products through brokers, but we were still using this green-screen IBM software.  ADRIANA: Oh, geez. [laughs] SHINGI: And there were like two customers left in Canada on it. And it got to a point where IBM was just like, "Listen, there are two customers on this platform. It just doesn't make sense for us to maintain it as a going concern for two customers. So you've got like nine months to figure things out before we sunset this." So the organization kind of freaked out because they'd never done anything in less than a year [laughs] from a technology standpoint.  ADRIANA: Oh my God. SHINGI: So no matter what the undertaking was, it just wasn't possible. The default answer was always one year plus. So everyone kind of freaked out. And this served as a bit of a catalyst to think differently about how we would approach this. So at that point in time, all of a sudden, people were bought into the idea of, like, well, why don't we try this agile thing in terms of how we approach this project? Because if we do this the normal way we do it, we know what's going to happen, and we can't afford for that to happen because it's our point of sale system. So that was a massive success.  And then you would think the company would rethink how we structure our technology delivery capability and build on the lessons learned. That's not what happened. [laughter] We just went back to how we used to do things. [laughter] It was like, yeah, that was a one-off. It worked, and it was nice. And yeah, we went back to business as usual. But by that point in time, I was kind of puzzled because I was like, okay, so we tried something different. Clearly, it worked. And there's some value to thinking differently about how we do things from a technology delivery perspective. So why would we switch back to how we were doing things?  Yeah, so I got a bit frustrated, and I left and started pursuing more work from a transformation perspective to be part of thinking differently about how we deliver technology solutions. What set of conditions makes for better outcomes? What sort of processes? What kind of context do you have to create around that? So that really became the area of interest for me. And that's kind of been the theme in terms of the type of work I've done ever since. ADRIANA: Cool. That's awesome. Having done a lot of this tech transformation type of work, what do you see are some of the commonalities around this in terms of what are some of the common pitfalls that you see a lot of these large enterprises getting into? And what are some of the common successes as well? SHINGI: One of the biggest pitfalls is thinking that it's all about getting off whatever our legacy technology is. It's all about getting off the mainframe, or it's all about getting on to the cloud.  ADRIANA: [laughs] ANA: It's true.  SHINGI: So, yes, it's important for us to modernize the technology infrastructure, but the thing is, you have to realize that the technology infrastructure is kind of like an iceberg in the sense that it's reflective of a whole way of working, and the mindset, and organizational structure. So you can't just plug out whatever technology you've got and think that you can plug in the new stuff, and it's just going to work. You have to think about it more holistically.  And usually, the other big thing, too, is that there's too much of a desire to be too aggressive out of the gate. So because the change is actually bigger than you think, it's better to kind of test the change in some small way in the organization to really understand what you're in for if you actually want to make an impact of the kind that you're looking for, typically, when people are talking about transformation. ADRIANA: That's a really good point. And do you find usually when you're working in these types of transformations, like, what's the appetite from leadership? Are you typically working with leaders who are super gung-ho about this? Or are they kind of being voluntold by other leaders? Like, what's the climate like? SHINGI: It varies. And it depends on what the narrative is around the transformation. So that's a big driver in terms of the behaviors that you're going to get and the context around it as well. Like, is it a new leader coming into the organization, and they're driving a certain kind of mandate? Or have we had something bad happen, and now we're really feeling the pain, and we need to make a change in terms of how we've been investing in our technology? So depending on what that context is, you get some very different patterns in terms of how things play out. ADRIANA: So what's the bigger motivator? Is it the new leader who wants to, like, let's change things completely? [laughter] Or is it the organization who's like, okay, this ain't working; [laughs] we got to fix it? SHINGI: Yeah, I think the new leader scenario is particularly challenging in the sense that obviously anybody who's coming to an organization as a new leader wants to show their value and put their stamp on things. But at the same time, there are reasons why those problems have persisted as long as they have. I remember one organization that I worked for had had several leaders come in and out, and it was always the same thing every time. It was like, oh yeah, we're finally going to get off mainframe. [laughter] That organization is still on mainframe today.  ANA: Oh, my God.  SHINGI: This conversation has been going on for like 30 years, [laughter] literally.  ADRIANA: I don't think we'll ever rid ourselves of the mainframe.  SHINGI: [laughs] And also, like, that's the other thing, too, is to think about what the actual aims are because do you need to really get rid of mainframe? Like, is that the issue? Mainframes are not terrible in and of themselves. I mean, it's a very reliable technology that, in a lot of organizations, has been highly optimized in terms of whatever they're having to do.  ADRIANA: Yeah, that's true. That's true.  SHINGI: It's not a foregone conclusion that, like, oh, if we just get rid of mainframes, all the problems will resolve. ADRIANA: And if anything, there's a lot more complexity that's created around it. I've been in an organization where they were trying to modernize around the mainframe, so it's almost like they create a moat around the mainframes, what it feels like, where they, you know, new services come in, and they plug into, you know, they feed into the mainframe. And they create these whole levels of abstraction over the mainframe. And it's, I guess, in some ways, like trying to put lipstick on a pig [laughter] because no matter how hard you try, the mainframe is still there. [laughs] SHINGI: Yeah, I mean, it's tough to actually plot a path out of the dependency because, typically, mainframes are running critical systems. So biting the bullet in terms of actually starting to shift workload away from mainframes is a big deal. And are you the one who wants to be on the hook [laughter] for actually making the switch in a real, meaningful way? Because if it goes wrong, it's on you. ADRIANA: Yeah, it's true. It's true. And there goes your reputation. [laughs] SHINGI: Yeah. And the other thing, too, is the timeframe that's typically associated with moving away from that type of technology is usually pretty short because all of a sudden, someone or some group of people decided this is now THE thing to do. And it needs to get done yesterday because we're behind. ANA: Always. [laughs] ADRIANA: Yeah, yeah. It does feel like, in some ways, some of these deadlines are arbitrary. And then you got people scrambling for what seems to me like no good reason if they had just taken some time to understand the landscape. And I think these are some of the pitfalls that I've seen myself where a new leader comes in, and they're like, "We got to change all the things." And then they don't take the time to talk to the people on the ground to see, okay, why is it that things are the way they are? They just want to make the change, and they want it yesterday like you said. And then I feel like that ultimately leads to their demise and the demise of the so-called transformation program. [laughs] ANA: It's that idea of not doing the planning or prepping kind of like going in 100%. We're doing all this, but you never know why you're doing it. Is it really needed? Like, asking, I guess, meaningful questions. SHINGI: Yeah. And kind of gauging what's realistic in terms of moving forward with the transformation objectives and balancing the risk because there's a huge amount of risk when you're moving off of mainframe infrastructure. It is typically running critical functions for an organization. And it's not that you can't get there. It's just, yeah, if you're going in with this idea that, oh, in a year, I'm going to make it happen and the five other people before me didn't know what they were doing, [laughter] you're about to be number six. [laughter] ADRIANA: So true, so true. ANA: I've definitely gotten the career advice of, like, oh, make sure to call people from that company who have left in the last three months before you join any organization just to find out what were really the hurdles when they were trying to push change, specifically within that infrastructure space and modernizing and digitalizing things or cloud-native transformation. SHINGI: And the other thing, too, I would say, is, so there's one part around what you need to do from a technical perspective. There's a big part around the culture and the ways of working. So you need to actually move on a bunch of different dimensions simultaneously. Like, if you just are pushing purely from a technology perspective, you're not going to end up getting very far. Because, like I said, the technology itself exists in the context of the organization's culture. It exists in the context of the organization structures. It's entangled in terms of how the organization works.  So if you're going to try and actually change things, you need to take a systems approach. Otherwise, you're going to, at best, you'll end up with these pockets where you've been able to do some things, and the rest of the organization is kind of doing things the way they've always been doing it. And it's an outlier. It's a source of friction because now you've got this dual operating system going on. ADRIANA: Right, right. You've got the cool kids doing it the new way. And then the "old timers" quote, unquote, doing it the old way. [laughter] Why are these kids doing this newfangled thing? [laughter] SHINGI: Actually, I've got a great story on that where one organization I worked with had that type of split. And the issue that it ended up creating was that so you bring in new technologies. You got new people doing things in a new way. You still have a dependency on the systems that exist in the rest of the legacy systems in the organization. And you still have to plug into processes that the organization has that exist.  So what happened there was you kind of had this parallel operating system going on. And people came in and built stuff, did stuff, shipped code, and shipped product with enormous security holes in it that they would not have known to look for without having a better working relationship with the rest of the organization that had the knowledge around legacy systems. So, yeah, I think you end up incurring a lot of risk in that type of situation. Because it was like, oh, we know what we're doing. We're using the new stuff. You guys don't know anything.  ANA: [laughs] SHINGI: Don't worry; we got it. We don't need you.  ADRIANA: [laughs] SHINGI: So the other part of the organization was like, okay, fine. You guys know everything. You don't need us. I'm sure you can figure everything out. ANA: [laughs] ADRIANA: Oh, wow. SHINGI: All of a sudden, we had an incident which cost the organization millions of dollars. ANA: Ouchy. And that's the thing. Sometimes it has to get to that point for some organizations to realize a lot of things need to change. And it sucks that it has to get to that tipping point. SHINGI: Yeah, it can't have this old versus new dynamic. That's pretty toxic from a transformation perspective. ANA: [laughs] Old guard versus new guard, definitely. [laughs] SHINGI: Yeah. And it's admittedly hard to navigate, right? ADRIANA: Yeah. SHINGI: Because there's some degree to which you have to start moving the organization in a different direction and building some different capabilities, doing things in different ways. But you also have to respect the existing culture and the heritage, and you have to have this transformation aspect of what's happening fit with the rest of the organization. And that's where, too, I think there's a piece where you're going to have to tackle some hard things in terms of what will happen eventually as an organization continues to transform because people's roles will change; jobs will change; the number of people you need doing certain things is going to change.  Certain functions that you had used to create value won't necessarily create value anymore. Being really mindful of that and being transparent about that, I think, is really, really important. Because people are not stupid, like, people understand that as things change and evolve and as technology changes and evolves, that is going to change things, including the applicability of people's skills and the value of the role that they're in. But I think if the organization is really transparent and supportive in the sense that they help people find the best outcome for themselves, whether it's upskilling, or moving to a different part of the org, or maybe you want to leave the org because you're not really interested in the direction that things are going in, that's an enormously important part of the equation because then creating that kind of safety and having that kind of transparency allows people who have been with the organization to work with newer people who are coming in and bringing new perspectives and new skills, and who are pushing different ways of working. ADRIANA: That's really good advice because I've definitely seen that in certain organizations where you got a team that's been doing it this way for 20 years, and it's pretty cushy. So they want to keep doing it the same way. But the organization also has to, like, it just has to evolve because you just can't stay where you are.  It's so hard to convince people who see what they have is an extremely stable job, very predictable. And, now, all of a sudden, you're telling them, "You must uproot everything you know about your job." And if you don't give them that safety net, that stability like you were saying, they can be very antagonistic and resistant to change, and then that can jeopardize your transformation no matter how well-intentioned. SHINGI: Absolutely, because those people are in possession of really essential knowledge that you're going to need to draw on to actually move things from where they are to where you want them to be. You need them as active participants. ANA: And it also brings to that point of psychological safety within the workplace that we talk a lot about on the podcast episodes to where it's like, you want to just make sure that your employees are being heard and that they're not in that sense of left to fend for themselves, that transparency aspect. It's like, we're in this together. We're going to learn together. We'll get through it. You're in this journey of inclusivity and belonging. SHINGI: And let's be honest about it; all the outcomes are not necessarily going to be what people want. So I may not want my role to come to an end, but given where we're going, it will. So the question is, is there support for me to do something different? If I decide that I want to leave, is the organization going to be fair and generous? That being in place, so the psychological safety afforded by that type of transparency within an organization is essential.  I really like the Netflix philosophy around we have different people in different times to lend their skills in different ways. And if we get to a point where we now need a different set of skills, like, it's not personal. It's just the organization has evolved. It needs something different. We respect what you've done for us. We value what you've done for us. We're going to treat you very fairly.  So, yeah, I think that kind of philosophy is really important in terms of navigating these changes because the reality is there is, especially in any kind of significant change effort, there's going to be impact on people and how safe they feel is going to dramatically affect the transformation outcomes because you need those people to be on board, even if it sometimes results in them leaving the organization at some future point in time. ADRIANA: Yeah, yeah, that makes a lot of sense. And I think on a similar vein; it reminds me of advice...I've read about the same advice in two different places where coming in and telling people, "You're doing it wrong," is just going to antagonize them rather than taking the time and asking the questions, "Well, why do you do it this way? Why not this way?" is a much gentler way and provides a lot more psychological safety [laughs] than coming in guns blazing like, "Y'all, suck. [laughter] Let's change this crap." Like, nobody wants to hear that. [laughter] SHINGI: There's no greater trust destroyer than coming in and saying, "You were doing it wrong." [laughter] And honestly, the reality is that I mean, any one person that comes into the organization is a complex system. The amount that you don't know about the system versus what you do know, it's like you probably know, I don't know, like, 1% of whatever percent [laughs] versus what is unknown. So there's a degree to which there has to be some humility because someone's coming in with whatever their experience is, and their skills, and their perspective. But I think the biggest thing that someone coming in with a transformation mandate can do is listen because there are reasons for everything. There are reasons why all the decisions have been made. There are reasons why certain things have not changed. There are reasons why certain opportunities have not been harnessed by the organization.  So gaining a really good understanding of what those reasons are, and understanding that, and being able to lean into where you can harness the organization to focus on starting to change things is really key because it's not about coming in and doing stuff to the organization. It's about helping the organization focus on where we can start to change things and start to see the greatest benefit and how we can start to think a little bit differently, perhaps, than we did about what we can do or can't do. ANA: That's true. ADRIANA: Yeah, that makes a lot of sense. It reminds me of some work that I'd done a few years ago where my team and I were trying to do some, you know, modernizing DevOps processes or even introduce DevOps to these various teams. And we were doing, like, we were working with each team individually. And then we realized, oh, crap, these are actually part of a bigger picture.  And it wasn't until we understood the bigger picture that we were like, okay, we're approaching this in such a piecemeal way that there is no way that we can actually do right by the organization because there are all these other interactions that we're not even aware of. So all we were doing initially was maybe chipping away at part of the problem rather than looking at it from that holistic view. So, for me, that was a huge lesson learned. Like, take that time to understand the scenario that you're walking into, right?  SHINGI: Yeah. I think taking that systems view of an organization is always going to be critical in terms of finding success with transformation. Because if you look at things narrowly in terms of, oh, I'm coming in, and I think I know what the problem is, or I'm coming in, and the problem is obvious, or I'm coming in, and we just need to do X. ANA: I love that you brought it back to systems because that was actually what I have written down here as one of the questions. Like, what advice do you have specifically for technology leaders or just tech leads that are wanting to create a change but they don't know what those systems are or what those dimensions are? How do they go about it? Like, I know we know that culture is definitely one of them. But I think there's a lot more that goes into it that you can talk a little bit more about. SHINGI: There's a piece around the organization's culture. There's a piece around the organization's processes. There's obviously whatever technology is actually in play. There's a really important piece around the political dynamics that exist and the power dynamics that exist in the organization as well. So it's actually looking at all those things and how they fit together to better understand where you have opportunities and how you can best leverage those opportunities. Because sometimes, something may look obvious in terms of what the benefit would be from a purely analytical perspective. Like, you look at the technology. You look at the money, like, it all makes sense. But then, probably, there's this political dimension that you're missing. [laughs] And so even though everything makes sense on paper in terms of if we just did this, and here's the business case, and the money makes sense, well, yeah, there's this whole other dimension that you're not accounting for. ANA: The politics can vary very, very much per company. Like, yeah, the size matters, but when you're in a toxic workplace, it brings out very different systematic issues within the organization. SHINGI: And, honestly, in my view, politics is not a bad thing in and of itself. I mean, it really speaks to a process of how us as human beings figure out what we are thinking as a whole. That's what that whole dynamic is about. So it's actually essential because it's a big part of the mechanism in terms of aligning people's interests and getting them to feel like whatever is happening is meaningful for them in a way that matters. So yeah, you definitely ignore that at your own peril because that kind of political dimension creates buy-in. It creates alignment ADRIANA: Mm-hmm. And it's probably, like, in some ways, more critical than the technology itself. SHINGI: Yeah. Without that, you can try whatever you want to try. You can spend however much money you want to spend. You can push as many mandates as you want. But if you do not have the right people on board in the right way, it will not work.  ADRIANA: Yeah, yeah, absolutely.  SHINGI: And there are always going to be people who are more influential in organizations than others. And it's not based on the org chart, by the way, like; that's the other thing that's really interesting. There are a lot of people in organizations that are influential that you would not necessarily know by looking at the org chart that these are key people within this ecosystem. And if we're going to kind of get any momentum with what we're doing here, these are people I need to make sure to win over or make sure to have as part of the equation. ANA: That concept of like social capital, like, those are the movers. Those are the ones that could advocate or could actually totally block something from moving forward. SHINGI: Yeah. And we've probably all experienced this, right? There are certain people that kind of have this type of influence in an organization where people are looking to them to see how they react before they decide how they're going to react. ANA: And the silent looks. [laughs] SHINGI: Yeah. And you've probably seen it in a physical environment, right? You can actually sometimes see it when people look over to a certain person to kind of get a read [laughter] of like, hey, how are you feeling about this thing? So depending on how their facial feedback is presented back from them, then they decide, okay, looks like you are feeling good about this, so I think I'm safe feeling good about it too. [laughter] Versus, oh, I don't think you like what is being said here at all, so I'm feeling very hostile. ANA: That's very true. ADRIANA: Yeah, yeah. It all goes back to doing that research up front and understanding your key players, your key challenges. SHINGI: Yeah, talk to people, understand what the organization already knows, understand the relationships, and understand the dynamics in terms of what's happened and why and the context in which decisions have been made. ANA: When looking at some of the work that you've gotten a chance to do in your career, what have you found that has worked as far as bringing agile into an organization? SHINGI: The most important thing is being clear on what the organization's outcomes are because Agile is an enabler. It's not a goal in and of itself. If the conversation you're having is like, we want to do agile because we want to be agile, that's the point at which I would now start to dig deeper to try and understand what are the actual outcomes? Because agile for agile sake doesn't buy you anything. ANA: [laughs] SHINGI: It's really about what the organization is looking to solve for, how it's looking to change, and the degree to which that agile toolkit can be of service. The way I think about it in my mind is like, if I call an electrician to my house, obviously, I'm having some kind of electrical problem. The electrician is going to be very knowledgeable in terms of how circuits work, and how electricity flows, and different things that can go wrong for different reasons.  So the first thing I expect when they show up is not that they're going to start talking to me about how electricity works, and circuits, and all this stuff. Like, I'm expecting that they're going to ask me, "Okay, what issues are you having?" "Oh, the power keeps flickering in a certain part of my house." "Does it happen all the time? Does it only happen some of the time? Is it connected to other appliances being on in other parts of the house?"  So they're coming in with a problem-solving mindset. They're trying to diagnose what the issue is, and they are working in the background through their knowledge base to think about, okay, based on what you're telling me, my hypothesis is it's this type of issue. Here's what I'm going to do. So after they actually diagnose the problem and solve it, then I may become interested in, like, oh, how did you know to think about that? Or how can I avoid this type of problem in the future? Or I'm just curious about how this works.  But that will only happen after you solve my problem. If you come in and you start trying to educate me on how circuits, electricity, and wiring works, [laughter] I'm not interested because I have a problem. I need help with the problem. And you are not helping with my problem. ANA: [laughs] ADRIANA: Right. Right. Yeah, yeah, it makes sense.  SHINGI: It feels like you're becoming another problem [laughter] in the sense that you are getting between me and someone who could be helping me with this problem. [laughs] ANA: It's very true.  SHINGI: So, similarly, I think if you're coming in to consult from an agile or ways of working perspective, it's really about understanding what the organization's problems are, what kind of outcomes they're looking to create. And then, you're going to use your toolkit to construct recommendations and hypotheses that are going to start creating value with respect to those problems. And then you can talk more about the technical stuff if you want. ANA: [laughs] SHINGI: If there's interest in doing that. But first and foremost, it's about actually creating value from the standpoint of what the organization's problems are. ADRIANA: Yeah, that makes sense. As we wrap things up, I do have a question for you around, you know, you've seen your fair share of digital transformations. How long does it typically take to win the hearts and minds of organizations with these types of transformations? SHINGI: So I could give you the consulting answer, which is it depends.  [laughter] ADRIANA: I fully expected that. [laughs] SHINGI: The best way to kind of win people over is to demonstrate value. Find something in the organization that we can start working on in a focused way that proves out the hypothesis in terms of whatever we want to change, and then build on that success because it's only a small percentage of people that are going to connect with a conceptual framework around how if we make changes ABC, we will have benefits XYZ. Most people need to see things happen. They need to actually be able to see it, and not only that, they need to be able to see it in the context of the organization. So yes, okay, we've seen it in the context of some other organization, great, but it'll never work here. So you need to actually prove it out in the context of the actual organization and then be able to tell that story and leverage the people who live that change as ambassadors to build more momentum and appetite to do other changes or to go bigger with the change. ADRIANA: That makes sense. So you give the rest of the org FOMO. [laughter] SHINGI: Yeah, that's the best way to do it, right? So, essentially, the change is spreading itself, right? ADRIANA: Yeah, yeah. SHINGI: Like, you've planted the seed somewhere in the organization. You've been able to do some things that have demonstrable value. You've got people who are talking about it, and then you're going to get some other people who are interested because they don't want to be left behind or some other people who are interested because they're like, you know, what? We're not very different from that other group. And if this is working for them, it could work for us. So then the change starts to spread pretty organically in an organization, but finding that first point is critical. Like, the big mistake, I think, would be to try and push through a big change agenda on just concepts. That's a really hard sell. ADRIANA: Yeah, yeah, yeah. Making that meaningful yet not necessarily huge change is very different from the concept of what you were saying earlier, where you've got an organization that has kind of a rebel offshoot that does things in a different manner. Because the idea is rather than the intention of them being different from the rest of the organization, this is meant to trickle down the desire to change. So it's less an us versus them rather than a come join us sort of mentality. SHINGI: Exactly. And in terms of that balance between the old and the new, it's better to augment an existing group with some new people who can help in terms of whatever the experiment is that you want to run. And then it's we did it together; there isn't this semblance of old and new rather than, okay, I'm going to get this group of people who are like all the good, new, shiny people who know things versus these dumb, lazy people. [laughter] ADRIANA: So, as we wrap up, do you have any final parting words for folks getting into, I guess, both for folks who are working in the digital transformation space and also for folks who have digital transformations imposed upon them? [laughter] What is your advice? [laughs] SHINGI: For me, one of the biggest realizations over time is that technology is fundamentally a people business. I know that people will think in terms of code and hardware, but it's actually a people business, and more so today than ever because... ADRIANA: Totally. SHINGI: Many of the things that people are working on from a software perspective are always going to involve large-scale collaboration in most big organizations, so it's a people thing. So with that being said, then it's really all about looking at things from that humanistic perspective and being able to get people aligned, creating the best environment to facilitate collaboration. Psychological safety is an important enabler in terms of being able to do those kinds of things. So, yeah, that human dimension is the biggest thing that you have to focus on. Everything else will take care of itself if the human dimension works. We'll figure it out. There are no insurmountable technology problems. ANA: I love it when you said where it's like a new person walks into an organization, and that's a complex system walking in. And it's like, words of wisdom. [laughs]  Well, with that, thank you very much, Shingi, for joining us in today's podcast. We loved having you on.  Don't forget to subscribe and give us a shout-out on social media via oncallmemaybe. And be sure to check out the show notes on oncallmemaybe.com for additional resources and to connect with us and our guests on social media. For On-Call Me Maybe, we're your hosts Ana Margarita Medina... ADRIANA: And Adriana Villela. Signing off with... SHINGI: Peace, love, and code. ALL: Whoo!

About the guest: Alex Hidalgo is the Principal Reliability Advocate at Nobl9 and the author of "Implementing Service Level Objectives." During his career, he has developed a deep love for sustainable operations, proper observability, and using SLO data to drive discussions and make decisions. Alex's previous jobs have included IT support, network security, restaurant work, t-shirt design, and hosting game shows at bars. When not sharing his passion for technology with others, you can find him scuba diving or watching college basketball. He lives in Brooklyn with his partner Jen and a rescue dog named Taco. Alex has a BA in philosophy from Virginia Commonwealth University. Find our guest on: Twitter LinkedIn Mastodon Find us on: On-Call Me Maybe Podcast Twitter On-Call Me Maybe Podcast LinkedIn Page On-Call Me Maybe Podcast Mastodon On-Call Me Maybe Podcast Instagram On-Call Me Maybe TikTok On Call Me Maybe Podcast YouTube Channel Adriana’s Twitter Adriana’s Mastodon Adriana’s LinkedIn Adriana’s Instagram Ana’s Twitter Ana's Mastodon Ana’s LinkedIn Ana's Instagram Show Links: Nobl9 SRECon Chef HugOps Admeld Customer Reliability Engineer (CRE) Service Level Objective (SLO) Squarespace Implementing Service Level Objectives: A Practical Guide to Slis, Slos, and Error Budgets OpenSLO OpenSLO Slack Community Sloth O'Reilly Google SRE Books Rundeck Break Things on Purpose Podcast - Alex Hidalgo  Additional Links: Google SRE Book - Service Level Objectives    Transcript: ANA: Hey, y'all. Welcome to On-Call Me Maybe, the podcast about DevOps, SRE, observability principles, on-call, and just about everything in between. I am your host, Ana Margarita Medina, with my awesome co-host... ADRIANA: Adriana Villela. ANA: Today, we're talking to Alex Hidalgo, who works over at Nobl9 doing all things Principal Site Reliability Engineering. We're very excited to have you join us. ALEX: Thanks so much for having me. ANA: So, to kick us off for today, we love asking our guests, like, what are they drinking for today's podcast episode? What's keeping you hydrated? ALEX: Water that I put in a refilled bottle that I got on the airplane flying home from SREcon a few weeks ago. ANA: What about you, Adriana? ADRIANA: I've got green tea. For once, I have something different. I always have water, so yay.  ANA: That is funny because I'm usually the one with the fun drinks. And today, I'm on water because I just forgot to grab anything different. [laughter] There's that whole, like, constantly thinking about too many things and having to hop on between meetings. It's like, what do I need for my next meeting to have near me? Like, do I need my notepad? Do I need my water? And if you're moving around your office, or your house, or commuting, it just gets a lot. ALEX: I always forget water half the time when I do these things because I have a neighbor who's actually the founder, and composer, and conductor for the Brooklyn Symphony, which is really cool. So he has this studio in the basement that he lets me use on occasion for stuff like this. And it's great because it's got pretty good sound quality.  And my dog, Taco, he's prone to bark in the background for no reason. But that also means I have to take everything apart, like my mic, and my webcam, and my USB dock, and my laptop, and throw it all in a bag and rush downstairs because I probably just had a meeting. And then half the time I get down here, I'm like, I'm about to talk for a very long time, and I forgot to even bring water. But today, we're all set. ANA: Yay. It's like rescheduling. And then how many restarts have we had to do of computers [laughs] and browsers? ADRIANA: We're so happy we finally made this happen today. [laughs] ANA: It is kind of nice because it does bring to the front what we love talking about, that reliability aspect of stuff. And everyone kind of finds their passion for reliability in a very different way. For me, I kind of stumbled upon it, and it was like, oh, this is really cool. But I know that Alex has a really interesting story and how they got into their career path. So I would love to hear from you. ALEX: So yeah, I'll try to keep the beginning part of the story short and gloss over a few things. But I was into computers from a very early age. My dad started teaching me how to program in BASIC when I was about nine years old. And then, in middle school, my friends and I taught ourselves C and eventually C++. And we decided we were going to write our own video games. We learned OpenGL and how to make 3D trains and things like that.  And I just maintained this passion for computers throughout my high school years to the point that I actually chose not to go to college. I figured I could go get a decent tech job right out of school, and turned out to be mostly correct. I ended up doing network security work for the Department of Energy. I did well. I actually got promoted within just a few months. I mean, I started doing overnight nighttime monitoring. I was watching a computer screen, watching potential alerts come in, trying to decide whether or not these were actual attacks on the network or just false positives, and most of them were just false positives.  And I did well, but after about a year there, I realized I kind of hated it. So I decided, you know what? Maybe computers aren't for me at all. Maybe they're meant to be a hobby, and I'm not meant to work with them for a living. And I realized, you know, I was still pretty young. I could still pack up everything and go to school, and so that's what I did. And I ended up studying philosophy and history. Then after that, spent my 20s working all sorts of jobs, restaurant work, front of house, back of house. I worked in a warehouse for a little bit. I sold furniture. I made most of my money as a DJ for about a year.  ANA: Wow.  ALEX: Just all sorts of random odds and ends. Then a whirlwind of circumstances landed me in New York City in early 2009. And so the 2008 recession, that downturn was still really, really in effect. And even though I now had this shiny degree in philosophy and history, right? [laughs] Very highly marketable. [laughter] I wasn't sure what I actually wanted to do. And so I started to apply for all sorts of random jobs. I thought maybe I wanted to work in publishing, for example. And my money was running out. I didn't have a lot of money. I was able to stretch it a few months.  And suddenly I was like, you know, I really need a job. And I met these two dudes who were at a bar one night, and they were crushing tallboys of PBR on their foreheads. [laughter] And so I went over to say hi, and turns out one of them needed to hire someone new at their IT firm, just kind of like a help desk-y small to medium business support kind of place. And I was like, you know, I can still do this computer stuff because it always remained a hobby for me. And so I said, "Sure, why not?"  And I went in, and we did a very brief interview. And I started just a few days later. And that was great because the day I got my first paycheck, I was using quarters and change. That's all I had left. Like, literally, I was down to change to go buy a cheap Bodega sandwich and to ride the subway. I'd just made it. And doing that job, it was help desk support. But also, I got to do some Linux work and networking work because some of our clients had those kinds of setup. And no one else had this...it was a 10-person company, like, no one else knew that aspect of things.  And I realized I actually did like working with computers. The thing I didn't like was working for the government as a 19-year-old. That was the thing I actually hated way back when. And so I did that for a few years. And then, I ended up moving to a company called Admeld as what we call the technical operations engineer. This is kind of before we consolidated on titles like SRE and things like that.  And it was a really cool company to work for. They were pretty early adopters of Chef. They were pretty early adopters of the true DevOps approach to things. Suddenly, I was learning what DevOps meant, learning about HugOps. I was learning about blameless culture, which is something I had not really been familiar with in most of the jobs I'd worked [laughs] up until that point. It was great.  And then, not too long after, Admeld got acquired by Google. And so suddenly, my title went from technical operations engineer to site reliability engineer. And I was like, what does this even mean? So I spent the next few years still supporting the Admeld platform because we were making money, right? Like we couldn't just turn it off. I didn't even really have a chance to learn about true SRE principles for the first few years that I was at Google.  I got to learn a ton more about different kinds of tooling. And my coding skills went way up. And it was so a great formative time for me. But then it came time to turn Admeld off. I was the last SRE on the team. Everyone else had transferred already. I was the last one standing. And I got to run the Chef knife ssh command that logged into every single server because we were on physical servers, I think, like 1,500 or 1,800 of them, something like that. And I got to run that final command that shut them all off at the same time. ADRIANA: Oh wow. ALEX: Like, that was a great feeling. [laughter] But then, after that, I transferred on to some other Google SRE teams, spent about two years on each. I spent a few years on managed systems. I spent a few years on prodmon, the production monitoring team. Then I spent a few years on CRE, the Customer Reliability Engineering team. And all this time, I learned more and more about true SRE principles, especially things like SLO and proper incident management, and things like that.  I even ended up starting traveling all over the world, teaching other SRE at Google how to do their jobs. Like, this is where I found my passion for writing, and education, and sharing the things that I've learned, especially on the CRE team, where a big part of the job was essentially teaching Google's largest cloud customers how to SRE, everything from pure educational effort to actually sitting down with them and looking at the architecture or looking at the codebase sometimes even. I mean, like, here's how we can improve. And that's where my true love of SLOs started. At prodmon, we actually adopted them and adopted them very well. And so, I was very familiar with what an SLI was and what an SLO was, and I got it. Like, it made some amount of sense. But on theory, we decided that in order to properly engage with all these different companies, many of whom were not even in the quote, unquote, "tech sector" at all, major retailers, and things like that, that we needed a common vernacular.  We needed to be able to communicate with each other correctly and know what the other one was saying. And we decided that that language would be SLOs. So a huge part of any initial engagement with one of these CRE customers was, first, let's teach you what SLOs are, and then let's help you set them up. And then once we have that going, once we have service-level objectives that talk about the reliability in a holistic and meaningful way, then we can help you become more reliable because that was the whole point.  So yeah, that's when I fell in love with SLOs. And I moved on a few years later and spent some time at Squarespace, where I was still focused on SLOs quite a bit, was able to get them to adopt them, and eventually ended up accidentally writing a book about them. And that's kind of why I'm at Nobl9 today because we're a company focused entirely on helping you do SLOs better. We're a company focused entirely on providing you with the appropriate tooling so that you can do them in the best possible way because not everyone has that tooling.  And a lot of monitoring vendors they let you do SLOs in some very basic ways but not with all the bells and whistles. So yeah, I somehow ended up spending the last about six years of my career focused almost entirely on service-level objectives. That's how I ended up, well, here today. ANA: It's such a fascinating journey just because there was that where you were already into technology. And then the whole detour to study philosophy but getting a chance to then take all these skills that you take from liberal arts, education, and humanities and just talking to people and communicating that you get to put it to work and education with your customers and writing books. ALEX: Yeah, I mean, I actually think...I like to joke... So I majored in philosophy; I minored in history. And I was actually just like two or three classes away from a minor in creative writing as well, so English. At the end of the day, it didn't seem worth doing. I like to joke sometimes. I took the three most useless degrees: philosophy, history, and English [laughter] and combined them all into one degree. But that's not actually true. I found that the skills that I learned, not just studying philosophy, history, and English, the ability to synthesize ideas, express them clearly to others, learning how to write well, learning how to communicate well, that served me very well in my career. I also think my time in the restaurant industry, for example, is another really good example of learning service. You are a server. You're providing a service for someone. It's not a chance that that's why we call our computer servers servers, and that's why we call our services “services” because they're servers that provide a service for someone.  And you really learn how to think about these things holistically when that's your entire job when your job is just to provide a service for these clients coming in. And you want to serve them reliably, even if that's not always the term you're using. And even if you don't explicitly have service-level objectives, you kind of do.  If you're working in the kitchen, you want X percent of your dishes to go out without them coming back, and that's a service-level objective. [laughter] And the customer being happy with their dish is a service-level indicator. We just use different languages in different industries. But it turns out we all kind of know these things. They're all kind of inherent to the human condition, I think. ADRIANA: That is so cool. It's funny how, even though oftentimes...like, a lot of our guests have taken the so-called crooked path into their current tech careers. I'm a huge believer that everything that you've done in your life up until this point has led you to be able to do the job that you do right now and that if you hadn't had your past experience, would you even be as good as what you are doing now? So it's so cool that having the restaurant experience and even having the philosophy degree gives you different perspectives, different appreciation of things in a way that maybe if you'd done like a CS degree, you might not have had.  ALEX: Yeah, I very much agree with that. I think about it quite a bit. And especially in the SRE space, in the DevOps space, in the TechOps, production, engineering, platform engineering, whatever we want to call it, [laughter] everything's really kind of rebranded sysadmin. And feel free to @ me about that on Twitter. [laughter] But in that space, yeah, I think it's very often the case that people don't have traditional compsci degrees.  I remember one time I was on a team where I think we may have had one new grad who had studied computer science. But everyone else on the team...we had an ex-chemistry professor. We had a psychology major. We had, you know, essentially the entire team. And it was a great team of very talented SRE, and almost none of them had any kind of formal academic computer background. ANA: Similarly, I was in a team of four, and three of us were college dropouts. And it was just kind of like pretty rad to be like, hey, like, yeah, non-traditional path for the win. ADRIANA: Yeah, clearly. Hey, my degree is in industrial engineering. ALEX: Love it. ADRIANA: [laughs] ANA: You kind of beat me to one of the questions that I wanted to have for the podcast of defining SLOs and SLIs. I think the analogy with servers really does help folks kind of put it in perspective. But if you were to give a quick summary and how they relate to service-level agreements as well, like, what would that be for folks that are really not familiar with it? ALEX: Sure. I'll answer that in two parts. How do you want to set good SLOs? Or actually, I like to use the term meaningful SLOs because they're not worth anything if they're not meaningful if they're not telling you something. They need to be a signal that you can use. At their heart, they're just a codification of the concept of don't let great be the enemy of the good.  Don't strive for 100% because you'll never reach it. It's impossible. And you'll spend way too much trying to get there. Your humans are going to be way burnout trying to get there, and the amount of money that you have to spend trying to get there is, like, it gets difficult. So pick a reasonable target that your users can actually absorb. And here, a user can be anything from a paying customer, to another service, depending on your service, to a team down the hall. It's anything that depends on your service, whatever your service looks like, and whatever the shape of that is.  A good SLI is one that tells you what your users are actually experiencing. It doesn't matter if your monitoring tells you everything's fine, your time series. Kubernetes is reporting that all your pods are running. Well, I don't care about that if the users of your service are not having a good time. So you need to figure out how can I measure what's actually happening from my users' point of view? And then pick a reasonable target for how often can that fail before they're going to actually be upset.  The second part is about how do SLOs relate to SLAs. There are two different parts to that, I think. One is that they're very different because, in my opinion, SLOs are most useful to be used as decision-making tools. They're better data to make better decisions with, to have better conversations to say, "Okay, this is what we believe our service has been performing like; is that okay? Do we need to go address that? How many people do we need to put on this? Do we have to drop everything? Or do we just assign one person for one sprint to see if they can go clean some stuff up?"  SLAs are written into the contracts, pretty necessarily by their definition. They are an agreement. An SLO is just an objective. You can change it basically whenever you want, as far as I'm concerned. Like, as long as your users aren't upset with your new SLI or your new SLO, then cool, do it, change it. SLAs you generally can't change because, again, they're generally written into contracts. And when you violate them, it's not a decision-making tool. It generally means that you owe someone money, or you owe them something, credits, you owe them, you know.  So while they're similar in the sense that you are trying to measure the performance of your service, and they both use target percentages...and these target percentages often have many nines in them, like the most common number 99.9% availability, and you're like, whatever. They're very different tools in my mind. But if you do have SLAs, if you are working in a position where you're responsible for services that inform an SLA that may cause your company to owe someone something if you violate it, you can still use SLOs very effectively because you just set them with a more strict tolerance than your SLA.  So let's say your SLA is set at, again, we'll say, 99.9% availability of some API of some SaaS product. Then you can set an SLO, let's say, like 99.95 or even 99.98. And this gives you a signal. If you are violating your SLO, you better take action because you're going to be violating your SLA soon.  ADRIANA: Right. Right. ANA: It's like ringing that alarm like, oh no, the storm is coming, like, paying customer number four is about to hit up your CEO in their personal email and come to their door like, oh no. [laughter] ADRIANA: Yeah, I really like that concept as basically like the SLO informs, or it's like your canary in the coal mine for whether or not you're going to be violating your SLA.  ALEX: Yeah. Yeah, yeah, yeah, a bit.  ADRIANA: That's very cool. One thing that I was wondering about because you mentioned you work at Nobl9 like; my understanding is Nobl9 has been pushing something called OpenSLO. Can you talk a little bit more about that? Because I'm actually really curious about OpenSLO. ALEX: So, although we started it, it really is an open-source community project many contributors from tons of different companies. I encourage everyone to go check it out. And it's basically that's a specification language to ensure that people can define their SLOs, and their SLIs, and their error budget windows, everything that comes along with service-level objectives that we have a common way of doing that across the entire industry.  And so it's essentially mostly a YAML spec, but it's constantly being iterated on and improved to allow for more and more functionality and different alert methods like defining what fast burn means to you. Like, how quickly are you burning through your budget? And the goal is to be vendor agnostic and, in the worst-case scenario, allow you to more easily adopt either multiple tools or move from one vendor to the next.  It's still kind of like a work in progress, but it works for some systems already. For example, Nobl9 does not yet ingest OpenSLO YAML directly, but we have a tool that will convert your OpenSLO YAML to Nobl9 YAML that you can then apply to Nobl9. Or there's a project called Sloth, which is an SLO framework for Prometheus. So you can use your Prometheus data to calculate error budgets, and Sloth speaks OpenSLO natively.  I know that there are a few other projects like that. I don't know which ones I can talk about, and I don't want to mess up which ones are currently actually working on it and which ones are just expressed interest. But I would say just go check out OpenSLO. There's a Slack you can join if you're interested; all the maintainers, contributors, and even a ton of users hanging out there. It's a very friendly bunch. Just go check it out and join the Slack. ADRIANA: Awesome, awesome. Yeah, we'll make sure to include that in our show notes as well. ANA: Yeah, initiatives like that where we're getting a chance to bring together vendor-neutral standards is always kind of nice where it's like, let's just be able to speak the same language and educate one another and try to have meaningful conversations around topics that really do matter. ALEX: It was definitely at least partially inspired by OpenTelemetry. The huge success we've seen there, for example, we're like, we should be leading the way but working with the community, and figuring out how we can do this for service-level objectives. ANA: Much needed. Like, I know when I started learning about SLOs and SLIs, I was kind of like a little over my head. And then there weren't vendors applying it. It was just kind of like a concept within SRE principles; this is what you strive for. And, like, these are all the numbers and the dashboards, and management cares about this. [laughs] Don't let anything break.  ALEX: There are still so many people who come to us who sometimes even literally point at the SLO chapter in the first book and two of the SLO chapters in the second Google book. Yeah, we're talking about the O'Reilly Google SRE books here. And they're like, "I want to do that." [laughter] Like, I helped write one of those chapters. It's mostly philosophical. You don't have to do it just like that. You don't have to do it [laughter] exactly like that.  But that's the problem is most people learning about it get these chapters that are, I think, very convincing. I think they're great chapters, but they make it difficult to actually go and do, and that's part of what we're trying to solve there for sure, both with OpenSLO, so people have a better idea of how to just define these things in the first place as well as that's Nobl9's whole goal. We want you to be able to easily do SLOs without having to build a whole bunch of tooling yourself. ADRIANA: Yeah, yeah. I think that's such a great idea. And it makes so much sense also to codify your SLOs because, I mean, one of the things about SRE is codifying your infrastructure. And SLOs are part of that, part of your reliability story. So why not make that something that you codify, right? ALEX: Yeah. And it's YAML again.  ANA: [laughs] ALEX: But that's using everything else anyway [laughs] and so... ADRIANA: I don't hate YAML. [laughs] ALEX: I actually don't, either. ADRIANA: Yay. [laughs]  ANA: It's not the worst thing ever. ADRIANA: There are some people who hate YAML so much, but I don't mind it. ANA: It does get annoying when things are breaking. And it's literally just like one or two spaces or stuff like that. [laughter] But -- ALEX: Oh, I have a great YAML story. When I was first learning YAML, period, right? It was because I was trying to set up Rundeck. I was trying to set up a Rundeck instance, and Rundeck configs are all in YAML. And so I was trying to set up a few different jobs, like, a few different workflows. And so, I started with copying the example straight out of the Rundeck docs. And, okay, I put that into etcd/rundeck.  I start editing it, and no matter what I do, only the last workflow is being applied, over and over and over again. And I'm like, what am I doing wrong here? I leaned over to my coworker. He was always there to fix my dumb mistakes.  ANA: [laughs] ALEX: And I'd spent a day on this already. And he was like, "Well, you have them all in the same file. You don't have any separators." I'm like, "What do you mean?" He's like, "You see in the example, you see how there's three dashes in between the different workflows?" [laughter] And I'm like, "I thought that was decorative." [laughter] ADRIANA: Oh my God, that's awesome. ANA: I mean, I feel like that's the mindset when you've been a sysadmin that you kind of like put ASCII code on stuff. ADRIANA: [laughs] ANA: You're just like, duh, lines. You're making it pretty. ALEX: [laughs] ANA: This is just fricking, like --  ADRIANA: I love that. [laughs] ANA: Two weeks ago, with a Kubernetes release team, I had one of those PRs that I was like, I couldn't get right the indentation by like two. And it was just like a line break is all I needed.  ADRIANA: Oh no. ANA: And I was just going back and forth for like 36 hours on a PR. And I was like, I swear I have a job, and I know what I do. [laughter] And I'm confident. [laughs] ADRIANA: Don't you hate those moments where you're like, oh my God, what happened to my brain? [laughs] ALEX: I had something that...I told the whole story on the Breaking Things Podcast, the Gremlin one, so if you want to hear the whole story, you'll find it. But I once spent a week or maybe even two due to an incorrect end-of-line character, so go listen to that story. [laughs] ADRIANA: Ooooh. Oh, man. ANA: I've had those. [laughs] ADRIANA: I've had those too. Those are the worst.  ANA: Oh, we're pushing the industry forward by, like, oh, we're sharing our stories, and failure is going to happen.  ADRIANA: That's right. ANA: And like part of it is that human side where, as humans, we also make mistakes like that. And it's like; it's okay to have [laughs] spent 48 hours on just one end of file character that really destroys your entire code. ADRIANA: But you'll never forget. I'm sure now you're like, yeah, I am always going to be extra aware of end-of-line characters. ALEX: But, yeah, failure happens. And it turns out humans are okay with that as long as things don't fail too often.   ADRIANA: [laughs] ALEX: That's SLOs in a nutshell. ANA: The best, like Alex Hidalgo, approved summary of it. [laughter] ADRIANA: That's this podcast episode in a nutshell. [laughter] ADRIANA: I wanted to touch back upon the experience that you had on the CRE team at Google and some of the work that you do now with Nobl9 customers. What are some of the examples that you've seen of bad SLOs, bad SLIs, maybe back then? And what are the ones now? How are they comparing? ALEX: I'm honestly seeing a lot of the same mistakes: both instances, most, not all. But most of the people that we were either working with on CRE or most of the customers at Nobl9 are new to the process. They are learning how to configure and how to use SLOs for the first time. And the biggest gaffes I generally see is conflating availability with reliability; they are very different things. And two, trying to programmatically generate too many SLOs for too many different endpoints and too many different services. Too much data, and you end up with a multiple comparison problem.  The best SLOs are, at some level, artisanally formed. At some level, they need to be meaningful. They need to be created for a purpose because they have to give you a good signal. And I've seen people in the industry...I've also seen it just with friends at other companies. People I know from Twitter telling me stories of "Yeah, we tried to do SLOs, and we ended up dropping it as a company after a year because we didn't see the value."  And most of the time, those companies that don't see the value is because they're just slapping 99.99% availability on all of our APIs. What does that mean? What does that tell you? One, that target is probably too high; very few services of any can actually hit four nines. But beyond that, just slapping the same number on everything without understanding how does this service actually operate? How does it need to operate? This is exactly the kind of thing that leads people to not seeing the value and then eventually just dropping the whole thing, which is a real shame because when you do them right, SLOs are absolutely game-changing, absolutely life-changing. ADRIANA: So if you were working with someone who's getting started with SLOs, what would you give them as far as advice or easing into it so that they can start seeing the value of them right away? ALEX: I think the best advice I can give is just understand that they're not a thing you do. They are a different way of thinking about the reliability of your service and making decisions about the reliability of your service. They're not like an OKR that you check off and be like, cool, we have SLOs. Now we're done.  I generally refer to them as SLO-based approaches, SLO-based approaches to reliability because that's what it really is. It is a different system. It's a different way of thinking. It's a different way of gathering data. It's a different way of thinking about data. It's a thing you do for all time. You don't just set them up and walk away. You iterate on them constantly because something about the world is going to change. Your dependencies are going to change. Your codebase itself is going to change. The requirements of your users are going to change. We could go on and on.  So that means your SLOs are necessarily going to have to change with them. They can't just remain the same for all time. So yeah, that's the best advice I would give. Understand this is not a one-time project. It's not something that if your CTO comes to you and says, "We're doing SLOs now," that you just do them and then report back okay, we're doing SLOs.  ADRIANA: [laughs] ALEX: It's a different approach. It's a different way of thinking.  ADRIANA: Yeah, that makes sense.  ANA: I had two questions that I feel would be great to ask you about. How often should folks be reviewing service-level objectives? ALEX: The general advice I give is a lot at first and then less often later.  ANA: [laughs] ALEX: When you first set them, you may have just picked the absolute wrong values, right? [laughter] You may not have understood what your users actually need from you. You may not have understood the shape of your data. You may have thought you did, but maybe your metrics and your telemetry looks totally different than what you expected once you start measuring it in this way.  So, at first, look at them quite often. Later, I don't know, once a quarter, like, something that helps you think about them more. However, I will say operational SLOs; you probably want to look at their status pretty frequently. Make that part of your weekly team sync. Make that part of your monthly opex meeting. Make that part of...whatever kind of cadence you have. You do want to look at their statuses quite often. Even if they're not alerting you, you still want to make sure that they're not acting anomalously. You want to make sure you're not just running at 100 because that means either you're measuring wrong or maybe you should be setting a different target. So, yeah, you review them in that sense. I think it really needs to be on a cadence, whether that's monthly, weekly, quarterly. But you do also need to be looking at your SLO definitions, which I think is what you were kind of getting at. And that part is often at first, far less frequently later.  I think in the SRE book, we've got something like, you know, once a month at first, once a quarter after that, and eventually, you can get to once a year. You have no signals telling you that these definitions are wrong. But I take a slightly more nuanced view which is, you know, everyone's favorite phrase: it depends. It depends on what your numbers look like. It depends on what your tooling looks like. It depends on what user journeys these SLOs are even matching.  Are these purely operational SLOs for a single SRE team with five people? Well, then you can probably review them quite often because you probably have a weekly sync, or an operational sync, or a handoff meeting, or wherever you can be appropriately slot in. But if it's like a cross-org, multiteam, multicomponent SLO that measures like, can a user log in, add an item to the shopping cart, and checkout with it? That you just can't review as often, and that's fine. So I don't think there should be any hard and fast rules besides that often at first, and then if you feel comfortable, less often. ADRIANA: I guess the moral of the story is, like, don't fall in love with your first SLO, right?  ANA: [laughs] ADRIANA: Because it's bound to change. And I guess it's also okay to make mistakes. As you said earlier, you might come up with a completely wrong SLO. And so, having that review process for your SLO becomes really important. Because I guess the other thing, too, is you might have had a set of SLOs that were working really well, and then you have a major release to your application. And then [laughs] your SLOs aren't cutting it anymore. So I guess that is also a signal to review your SLOs once again because they're not doing their job, right? ANA: Yeah. I love that you said just don't fall in love with your SLOs firsthand. And I think even just in general, like, overall, they're constantly going to be changing. ADRIANA: Thank you so much, Alex, for joining us in today's podcast. Don't forget to subscribe and give us a shout-out on social media. Be sure to check out the show notes on oncallmemaybe.com for additional resources and to connect with us and with our guests on social media. For On-Call Me Maybe, we're your hosts Adriana Villela and... ANA: Ana Margarita Medina. Signing off with... ALEX: Peace, love, and code.

About us: Adriana Villela is a Sr. Developer Advocate at Lightstep, based in Toronto, Canada, with over 20 years of experience in technology. She focuses on helping companies achieve reliability greatness by leveraging Observability, SRE, and DevOps practices. Before Lightstep, she was a Sr. Manager at Tucows, running both a Platform Engineering team and an Observability Practices team. Adriana has also worked at various large-scale enterprises in both individual contributor and leadership roles, including Bank of Montreal, Ceridian, and Accenture. Adriana has a popular technical blog on Medium, co-leads the OpenTelemetry End-User Working Group, and is a HashiCorp Ambassador. Find her on Mastodon at @adrianamvillela@hachyderm.io to talk all things tech. Ana Margarita Medina is a Staff Developer Advocate at Lightstep, where she speaks on all things SRE, DevOps, and Reliability, and is a podcast host for On-Call Me, Maybe. She is a self-taught engineer with over 12 years of experience, focusing on cloud infrastructure and reliability in the last few. She is also part of the Kubernetes Release Team (v1.25 - v1.27) and has been advising CNCF's Keptn project since 2019. When time permits her, she leads efforts to dispel the stigma surrounding mental health and bring more Black and Latinx folks into tech.  Catch her on Mastodon at @anamedina@hachyderm.io about traveling, diversity in tech, and mental health.  Find us on: On-Call Me Maybe Podcast Twitter On-Call Me Maybe Podcast LinkedIn Page On-Call Me Maybe Podcast Mastodon On-Call Me Maybe Podcast Instagram On-Call Me Maybe TikTok On Call Me Maybe Podcast YouTube Channel Adriana’s Twitter Adriana’s Mastodon Adriana’s LinkedIn Adriana’s Instagram Ana’s Twitter Ana's Mastodon Ana’s LinkedIn Ana's Instagram Show Links: AWS Well-Architected Framework - Sustainability Pillar Adrian Cockcroft on "Architecting for Sustainability" HashiCorp Nomad Tracetest Malabi Helios Trace-Based Testing HashiTalks 2023 Julie Gunderson Conf42: Don’t Forget the Humans with Julie Gunderson and Ana Margarita Medina Keptn LitmusChaos Chaos Mesh Argo Cilium Backstage Kubernetes Release Team Crossplane Adriana’s blog series on running ArgoCD on Kubernetes ESCAPE 19: Chaos Engineering in a Multi-Cloud World with Ana Margarita Medina OpenTelemetry End-User Working Group OpenPolicy Agent Andi Grabner Observability-Landscape-as-Code Terraform Service-Level Objectives Stephen Townshend Slight Reliability Episode 39 - The Future of SRE with Adriana Villela and Ana Margarita Medina Additional Links: CNCF Working Group on Environmental Sustainability Cloud Carbon Footprint Project on GitHub TechstrongTV - What 2022 Taught Us About SRE’s Future Transcript: RIAAN NOLAN: Don't worry about failure. Fail fast if you do fail. None of us are superstars or anything, so your name doesn't mean shit anyway. [laughter] So put your code out there; if people think it sucks, it sucks; if they like it, they like it. The best thing is just paint your picture. Do your thing, and put it out there because that will help you grow.  LIZ FONG-JONES: And I think this is an area where we, as developers of observability tools, can really help because an SLO has to be a living, breathing thing, not just a thing that you put up on a dashboard and you look at it 90 days later and, oops, we blew our SLO. NORA JONES: But we're also all engineers. We've been a part of the big technical aspects of it all. And we've seen the social aspects not really be spoken about very much. I think it's a big miss. And it's honestly a business advantage to be able to talk about the social aspects as well. So we're really trying to give every company that advantage. ADRIANA: Hey, y'all. Welcome to On-Call Me Maybe, the podcast about DevOps, SRE, observability principles, on-call, and everything in between. I am your host, Adriana Villela. And with me, I have my awesome co-host... ANA: Ana Margarita Medina. ADRIANA: And today, we are kicking off Season 2, and I'm super, super, super excited to be doing that. We've had a little hiatus over the holidays, and we've had time to get refreshed and revitalized. Ready to take on 2023, hopefully, knock on wood. How are you feeling? [laughter] ANA: I'm hoping that the year continues to get better and that this year is better than last year, and that we see more amazing things happen in the industry, but that everyone takes care of themselves a little bit more. ADRIANA: Yeah, cheers to that. Now, in proper OCMM fashion, we must ask each other what we're drinking. So, what do you have today? [laughs] ANA: This morning is my famous Guayakí's organic Yerba Mate in enlighten mint flavor. You'll probably find me either having a latte or one of these as I kick-start my day. So we're still rolling out here in the Bay Area. We're trying to get the engine running for today's day. How about you, Adriana? ADRIANA: I've got green tea. I made myself green tea just before we recorded because I figured I need something a little bit more interesting than the water. I mean, water is great, and drink your water, but green tea today. It's kind of a dreary, rainy day in Toronto today, not compared to the rain that y'all have been getting in California, so big hugs to you. Yeah, every day in the news, there's new stuff [laughs] about the rain. I'm like, ehh. [laughs] ANA: There's definitely a big hug sent out to California. But I feel like there are so many places in this world right now that, with global warming, they're suffering from such terrible conditions of weather. The amount of flooding that I got to see in pictures and online and in parts of California was just mind-blowing that it can happen. But it's just a reminder that we take care of the world.  And even though this podcast focuses on reliability, sustainability is equally as important. And maybe this season, we will get a guest that talks a little bit more about that, or next season. Or if y'all want to chat with us on social media, we're always happy to hear folks' thoughts on it. ADRIANA: Yeah, absolutely. And also, a note for our guests: we now have a presence on Mastodon, TikTok, and Instagram in addition to our LinkedIn and Twitter presence. So you can find us on multiple social media platforms now. ANA: And you can even put in a request there of your favorite podcast guest that you would love to see be part of On-Call Me Maybe. ADRIANA: Yes. ANA: Questions that you want to ask any of our guests, questions for us, any feedback. We'd love to hear it. Let us know what you thought about season one or if you missed us between this break. ADRIANA: Oh, you know, I want to circle back to what you were saying about sustainability and technology because I don't know if you feel this way. So I've always, like, the environment has been near and dear to my heart since I was really young. And I kind of feel guilty working in tech knowing that the type of work that we do contributes to a certain extent to environmental problems because when you think about things running in the cloud, you've got servers running all the time, consuming a crap ton of energy. And I feel a little bit guilty. And I'm always curious as to how there can be a marriage between what we do and environmental sustainability if there's sustainable tech. ANA: Hmm. I think that is an interesting question, like, is there sustainable tech? I think, as an industry, we are waking up to the conversation of sustainability slowly. And it has increased more than ever in the last two, three years. And once again, I think this is actually something to attribute to COVID. We've had more of a demand within the supply chain. We've also had slowdowns on air pollution due to travel, and folks are starting to realize, like, oh, if we were to make this change, our world is happier, our world is sadder. And I think that cause and effect made people be like, oh shit, my actions do matter.  And then specifically to technology, I remember...I want to say it was three or four years ago. If you follow Amazon Web Services, they have something called the AWS Well-Architected framework. It covers five different pillars of how organizations can actually be building their applications. This covers things like making sure that you have a reliability pillar, that you have a security pillar, that you have an operational pillar.  But two, three years ago, one of the vice presidents at Amazon Web Services, Adrian Cockcroft, created the Sustainability Pillar and created this entire sector within Amazon Web Services that was going to focus on reducing the footprint that they were imprinting in this world or just letting their customers be more aware of it. And I thought a big organization such as them taking that step is a step, and it gets the conversation moving.  And then it trickles down to what actionable things can companies do? Well, even starting to look at capacity planning is one way to move forward in the sustainability conversation. Like, if you have all of your servers constantly just using 20%-30% of your resources, maybe you shouldn't be running a fleet that large, and maybe you can make it a little bit smaller. And therefore, your footprint is a lot less, and you're damaging earth a little less. ADRIANA: Yeah, that's so true. That's so true. Because I think sometimes we get so caught up on reliability that we need to make sure customers are happy, systems are up and running that sometimes it can be easy to over-provision resources, especially when you have the cash to burn. ANA: [laughs] ADRIANA: Maybe not so much when you don't have the cash to burn. In that case, yeah, it's like...[laughs] and it's worthwhile doing an audit of your systems to make sure that, like, yes, I actually need all of that capacity versus, oh, shoot, I'm using like 10%-20% of my resources. ANA: Do you have any thoughts on some of the things folks can do to audit what their spend is? I know that there are consultants out there that would help you lower your cloud footprint bill. But when we're thinking about this audit, a lot comes to mind. Because I think this is actually something interesting to throw out there as we kick off the New Year, as folks are going into their jobs thinking like, what are we going to do to survive the economic atmosphere that we have but also bring in a positive value to this organization? ADRIANA: I'd say the biggest one for me is definitely making sure that you're using what needs to be used. A more maybe not so direct to cloud usage but even related to a certain extent to the type of work that we do, which does involve some travel, is reducing business travel. That's a huge one. I would say consulting travel that's a huge one. So when I started my career back in 2001, I worked at a consulting company, and it was expected that you were going to be traveling to work at client sites. You were lucky to get an in-town project. I was actually quite lucky that most of my time with the company, I worked on in-town roles. But my husband, who's still at the company, he spent, I think, the first ten years of his career traveling for work. When the pandemic hit and all of a sudden, everyone's working from home. To me, it was like, oh, well, this finally proves to folks that you don't have to travel for all the work things. ANA: [laughs] ADRIANA: So, to me, it's like, make those travel plans if they're absolutely necessary because a lot of the stuff you can accomplish just working from home. Even traveling to an office, if you're working from home, it means that you're not commuting to an office. So it means that you are not driving or you're not taking public transit, which mass transit is awesome, and I will take mass transit whenever I can. I personally hate driving. But even taking mass transit, if you don't need to take that mass transit to get to the office, you're already, I'd say, helping offset some of that carbon footprint compared to having to commute. ANA: That is true. And it kind of makes you wonder, like, what goal, what wish list can you put this on on reducing your footprint as an individual, as an engineer, as an organization? Like, what small step can you take today that could actually accumulate by the end of the year to be a pretty significant amount? ADRIANA: Yeah, totally. Another one that comes to mind is I remember at some workplaces; I would always try to log in and out of...when I had a tower, I'd log in and out of my desktop power off for the day. But some companies will put so much crap on your machine that the process of booting up your machine and shutting it down for the day is so lengthy. I remember for a while, at some companies, I'd just leave my machine on. It kind of defies logic that there's so much crap on your machine that now you're compelled to leave it on to get a little extra bit of productivity, so even something simple like having the ability to power off your machine. I think with a laptop; it's a lot easier because even though I don't power my laptop off and on regularly, I don't always have it plugged in. I'll only plug it in when I absolutely need to.  My personal habit is I always turn off my power bar when I'm not using it because even though it does consume a small amount of power, it still consumes power. So it's these little, little things that you can do in technology to help offset the footprint. And if you've got enough people doing that, well, you can actually make a pretty decent dent. ANA: I 100% agree. I want to say that I don't always take a lot of these actions, but they definitely come to mind sometimes where it's like, it's that small thing of unplugging your laptop before a long weekend or any weekend. Or just kind of being like, you don't need to be connected to power to run updates, or even to just constantly be charging because I'm not going to be using you for a longer length of time.  But even like you said, I think it's actually a really nice idea to be like; this is the power strip for my entire work setup. When I'm not at work, the entire strip itself can be off. That's actually something I've always wanted to implement for when I travel; when I go away that it's like, "Alexa, power all of this off," or "Power strip goodbye." ADRIANA: Yeees. ANA: And we know that we are not consuming power, like, less risk of any fire emergency. And at the same time, you're just like, my devices are actually fully sleeping, not connected to the internet, not connected to anywhere. ADRIANA: Yeah, so you have some peace of mind, a little bit of mental health injected in there, too, right? ANA: [laughs] Are you telling me you're sending my devices on a mental health break? ADRIANA: [laughs] I think sending your devices on a mental health break sends you on a mental health break, [vocalizing] tan-tan-tan. [laughter] ANA: On that note, I'll always throw my little snippy where I say that if you travel with your laptop, you're not really going on vacation. [laughs] [laughs] ADRIANA: Yes, I totally agree. I've never gone on a vacation with my laptop, thankfully. I always make it a policy when I'm on vacation; I'm on vacation. I'm not going to respond to your Slack messages. You can text me if it's for fun stuff.  ANA: [laughs] ADRIANA: Otherwise, we're not talking, [laughs] and I will block your ass. [laughs] ANA: It's funny because this was just a conversation I recently had with my best friend around laptop going on vacation or not. And I'm one that says, "Don't take your laptop on vacation," and known to do it. And sometimes that I've done it, we've been on an island and having a laptop comes in clutch for locating cars, for locating things to do, restaurants, ordering, getting a new flight, and stuff. It's kind of interesting when you're planning your vacation, and you're like, well, last time I took it, but I told myself not to, but it came in clutch. But wait, I shouldn't do it, right? [laughs] ADRIANA: And that poses a conundrum because then if you brought a personal laptop with you instead of a work laptop, would it be less tempting to do worky things? Because nowadays, even though work laptops have a lot of the systems that you need to access to be able to do your job, a lot of times organizations have their email and stuff on the cloud, so therefore, you could technically access it from anywhere, which means that even if you're not bringing your work laptop, it can be super, super tempting to log into your email. Oh, let me just see what people are saying on Slack or emailing. [laughs] ANA: And, I mean, this is a personal preference on how folks handle vacation, I think. And it's a hard conversation because [laughs] I think I interject in this conversation quite often. Because then the same could be argued that they already have that access on their phone.  ADRIANA: Yes. ANA: So we might be agreeing no laptops on the trip for work reasons, like, no work laptops. And then it's like, let's not bring personal laptops so that we're not tempted to just fall into your habits: checking Slack, checking Twitter, and doing what you do at home, what you do at your office.  But then again, we forget that we might have one or two of the applications we use at work already set up on our phone. And just out of boredom, doomscrolling, like, waiting for your food, and you don't know what to do with your hands or time, and you end up checking your email. And you're just like, oh, crap, I'm supposed to be spending time with friends and family right now, oopsie. [laughs] So it's kind of interesting because, for some people, that doesn't bother them. And for me, I have to have 100% disconnect. I have to unplug that cable and walk away to feel like I'm actually recharging. Some other people don't have that need, and it's a personal preference. ADRIANA: And I think that's a really good point because I think it shows a lot of self-awareness that you've gotten to the point where you're like, no, this is what I need to stay healthy. And it's true; it depends from one person to the next. But having that level of awareness, I think, is huge because it ensures that you have a healthy vacation. Because you end up with a couple of problems when it comes to vacations, right? You get the people who don't take vacation because they feel guilty about taking vacation. And then you have the people who take the vacation, but then they're working through a chunk of their vacation, so is it really a vacation? ANA: Totally. It's always kind of interesting. And I know for me since we left you all last episode thinking about what we were going to do and what we were grateful for, I know I for sure was very grateful for the time off that I had and the people I spent it with and very much being able to disconnect.  I ended up being kind of bummed out that I didn't get to do as much cooking and Latin American traditions that I was hoping to incorporate during the holiday season. But when work kicked off, I felt recharged. I don't know what the holiday break was for you, Adriana. ADRIANA: Yeah, I kind of felt the same way. And I don't know if you're like this, but I'm always like, go, go, go. And if I'm not doing something, I feel guilty that I'm not doing something. And during the holidays, the first week was still kind of busy because it was like the week between Christmas and New Year's, so there was stuff to do kind of, sort of. And I decided to repaint my office, my home office, [laughs] so that took a chunk of the week.  But then, the second week, my husband and my daughter, and I just bummed out completely. Like, we didn't go out much, like outside even. We just bummed around and watched so much TV that we got to the point where we're like, "We've watched too much TV. Let's do something that's not TV." So then we started playing games, but it was super idle, but it was so good.  And I'm proud of the fact that I told myself, like, it's okay to give yourself permission to relax. Because what I always feel at the end of each year is it's been a hustle the entire year. And for me, last year was super stressful with losing my mom and stuff. So it was really nice to take that time to recharge, give myself permission to relax, knowing full well that when January hits, it's going to be all hands on deck because that's just my personality where I've got to be doing something; otherwise, I feel unproductive. So I'm grateful for that time to have been doing nothing, [laughs] especially that second week of break. ANA: [laughs] I relate there too. I know that my first one was very relaxing. And then my second one was more vegetating, and I very much needed it. I know you mentioned jumping back into work and it being pretty busy. Can you tease listeners with some of the projects that you're interested in working with or that you're already working with that folks we'll be seeing in the upcoming weeks?  ADRIANA: Oh yeah. So I guess some of the projects that I'm working on right now, I don't know if they're necessarily related to any of the recordings that we're doing. But I've got a HashiTalk coming up in February where I talk about How to Convert Kubernetes Manifests into Nomad Jobspecs.  I wrote a blog post about it just before the holidays, so I've turned it into a talk. So I've been hard at work on that. It's funny how even having a blog post with the script for a talk it's supposed to make things a little bit easier but converting from a blog post to a talk is a little bit difficult because you're going from all words to visual. So that's been kind of challenging.  And then this year, I'm super stoked to be playing around with trace-based testing. I've spent some time playing around with Tracetest, which I played around with it when it first came out last year. I think it was sometime in the summer. It was like an early, early version. Then I took it up again in the fall just before the break, I guess, where I had originally played with it because I'm like, oh, I wonder if this thing runs on Kubernetes. I wonder if I can run it on Nomad. So I'm like, that was my project in the summer.  And then I took it up again thinking, okay, what modifications do I need to make to Tracetest, like the new version of Tracetest, the Nomad jobs to get it to run on Nomad? And it ran pretty smoothly. And it's cool to see how much Tracetest has evolved even since the early days. There's a CLI, and you can define tests programmatically or declaratively through YAML, which has been super cool. So I really want to explore that.  And then the other cool thing is, you know, on the trace-based testing theme, I met with the founder of Helios, and they also do trace-based testing, but they take a different approach to it, which is really cool. Like, whereas Tracetest is more declarative, and you install the tool on your own cluster, whether it's Kubernetes or Nomad in my case, Helios is a SaaS product, and they've gone for more of an imperative approach.  So they've got a whole SDK for declaring your tests, and they support different languages. So I had a chance to speak with him last week. And I'm actually super stoked to try that out because, hey, what's better than one trace-based testing tool? Two. So hopefully, in the next few weeks, I'll have a chance to play around with that, and we'll see where it goes. So that's where my mind has been at.  And, of course, we have been hard at work interviewing folks for OCMM. And we're hoping also to bring content beyond just the podcast. So Ana and I have been brainstorming just fun ways to bring y'all content that's not just audio-based but also some fun visuals, so stay tuned for that as well. How about you, Ana? ANA: It's a very exciting next few weeks, I think, for you and for myself included. There are definitely a lot of amazing ideas going. And as you've shared, some of the work that you're doing with trace-based testing is very much exciting to see that space continuing to grow, considering it's something that was just a myth, an idea that folks were talking about, I think, in 2018.  And I think we're pushing the envelope. I think it is driving those conversations to think about how do we actually make some of those Day 2 operations a lot easier? And just make it easier to have the developer do more, care more, actually be an owner when it comes to their code, which is very exciting. I know for me, there's a lot going on always in my life and a lot of moving pieces.  So I have a few talks coming up; some of them will be talking about the similarities between caring about humans and caring about systems. So I'll be speaking at Conf42 with Julie Gunderson on Don't Forget the Humans. I'll also be keynoting some conferences around site reliability engineering and chaos engineering that have yet to be announced. So when those do come out, we'll be pushing them over on social media.  And as far as some of the other projects that I have going on for these next few weeks and this year, is that I'll continue being heavily involved with Kubernetes. I'm joining the release team of release notes team again as a shadow, so for Kubernetes version 1.27. And I'll be joining the team again, which is exciting. We just had the kickoff call last year.  And the release team is beautiful in terms of everything that happens behind the scenes to make all the Kubernetes versions go out. But being part of that program really allows for you to continue learning just how complex systems work and the human side of it, of course, by just staying up to date on the CLI portion of it, the storage aspect of it. Like, there's a lot to learn in Kubernetes, and I constantly feel like I know nothing, and I've been working with it for a few years. But I'm very excited to continue working with Keptn. I joined their governance board in December.  ADRIANA: Yay. ANA: So I'll be a little bit more involved in helping shape up what it means for this project to be within CNCF, along with partnering with other open-source projects. And as time frees up, I'm very excited to also be playing around with projects like Argo, and Cilium, and LitmusChaos, Chaos Mesh and seeing how they all come together. I really do enjoy that space of site reliability engineering and understanding your systems more and making them more reliable. And in my wish list for when I have more time and stuff, of course, I do have Backstage and Crossplane to revisit of just really being able to package things up as neatly as I want. ADRIANA: Yes. ANA: And really being like, oh, this is the POC of what I wish that I could do if I was starting an organization, an SRE practice from the start. So there's a lot of learning and content to be putting out there. And, hopefully, we get to teach folks a little bit more of how to modernize their applications, especially with those that are coming in from cloud-native environments only, that they're coming in from blended environments of being all bare metal that are slowly transitioning to the cloud, or folks that are completely like no access to the internet. How can they use some of these open-source technologies to start implementing and start going through a digital transformation within their organization? ADRIANA: Yeah, that's awesome. And those are really cool technologies that you've mentioned. And I can't wait to see what comes out of your explorations. I think you and I were chatting the other day about Argo because I got my start, I guess, on my blog by blogging about running Argo CD on Kubernetes. And I have not touched Argo in a few years.  ANA: [laughs] ADRIANA: So I would love to see how it has evolved in the last little while. And Crossplane, I think I played with it back in 2021 when it was still pretty new. And there wasn't a lot of support for creating resources in GCP, there was some. That's another really cool one that would be fun to revisit. ANA: Same. I think the last time I touched Crossplane had to be around 2019. I was speaking at ESCAPE/19. It was a multi-cloud conference. So I was trying to use Crossplane to do my demo. And after various hours of getting an almost demo working, like, what I was trying to do is just how do you leverage chaos engineering to build out your multi-cloud strategy? Like, making sure that you have latency in mind or that you're thinking about failover in between the clouds and all these little, tiny details that we were having to think about while I was working at Uber.  So looking at Crossplane, to me, was really interesting because it was kind of like breaking the fold in the industry in that sense, like, not many folks were talking about these spaces. Now that it's been a few years and we've seen the growth of Crossplane, we've seen their community be pretty successful. And a lot more tutorials, a lot more stories are out there. It will be pretty insightful to pick it up and see how it all ties together with all the other advanced open-source projects that we also have out there. ADRIANA: Yeah, absolutely, absolutely. I think it's going to be a really exciting year. And I also want to make a little plug for OpenTelemetry. Since we're talking about various open-source projects, one of my goals for 2023 is to be even more involved with OpenTelemetry. And last year, I was super excited to have made my first-ever pull request into OpenTelemetry. And I've made a few contributions throughout the year, which is exciting.  So this year, I also joined the OpenTelemetry End User Working Group, which I'm super stoked for because I think this is such a great opportunity to hear stories of real people using OpenTelemetry in real life. So I think it's going to be super educational. I hope that we can learn a lot and share a lot of stuff with the community around that.  And my call to action is anyone who loves using OpenTelemetry and is interested in getting involved; there are so many different ways of getting involved. You can write a blog post for OpenTelemetry talking about how you've used it in your organization, contribute to the docs, make contributions to the OpenTelemetry demo app, which I think has blown up since it first came out in the summer of last year. Now there's like so many different services, so many different moving parts to showcase how cool and awesome it is.  Or if you want to, just join the OpenTelemetry End User Working Group and share your learnings with the rest of the community and take that as an opportunity to learn from others as well. Those are some great and easy lower barrier to entry ways of contributing. ANA: Oh, man, how can I forget about the OTEL project? I was thinking about all the other open-source projects that I'm so eager about. But getting a chance to be involved in OTEL last year was pretty exciting. I've always tried to stay a little bit away from observability because I always felt like I wasn't going to understand it too much. And last year, I did get a chance to do contributions to projects around OpenTelemetry, and that was pretty exciting.  But as we think about all these other site reliability engineering projects that are coming together, it's also going to be exciting to see them all roll up into larger initiatives that we get to put together. And I know that through the time that we've been working on On-Call Me Maybe, working with some other guests, one of the other things that has come up for Adriana and I is very much in, like, what is the wish list that we have for SRE going into 2023?  I think, for me, a lot of it is making sure that we are understanding our system. And the way that I see us understanding our system, of course, is by leveraging observability first. And then, as we do that, making sure that we go back into those principles that infrastructure as code taught us. And we codify everything that we can, including those service-level objectives.  ADRIANA: Yes. ANA: As you're setting up those service-level objectives, make sure that they have in mind reliability goals that you're practicing, injecting failure, embracing the failure that you have, that you pause and you learn from that failure and that you're ready to make changes, but that you only do it when it makes sense for your team. Not everyone needs to go and change their entire culture because a brand new SRE book came out. ADRIANA: Yes, that's right. It's the fit-for-purpose SRE. Do what works for you. ANA: What was on your SRE wish list? I know you, and I have a lot of similarities. But I know there's other stuff that you also think about.  ADRIANA: I think the underlying theme that you and I have hit upon is really honoring SRE and DevOps principles by codifying all the things which I'll make this a plug for our Observability-Landscape-as-Code work that we started last year where we're basically treating the observability landscape as things that can be codified, which includes...it's not an exhaustive list, but it includes instrumenting your application, having a means of sending your telemetry to your observability back end, so configuring like an OpenTelemetry collector programmatically, whether it's Kubernetes or whatever, like via Terraform to deploy your collector somewhere. Codifying SLOs, which is something that I'm hoping that we can hit this year as part of our wish list. I feel like I'm missing stuff because I'm doing this from memory. Oh, being able to configure an observability back end through code, which is like part of a little demo that Ana and I did last year where we created dashboards in Lightstep through Terraform.  I think most observability backends have some Terraform provider that allows for configuration, so we really want to expand on that. So keeping that theme codify all the things, I want to see greater adoption of policy-as-code. I think there's OPA, Open Policy Agent, out there which enables that. So wouldn't it be nice to see greater adoption of that sort of thing?  And, of course, trace-based testing is big on my wish list. I'd like to see more of that, whether it's through...whatever tool you choose that's out there, Helios, Tracetest. There's another one called Malabi. I think bringing testing into both developer concerns and reliability concerns is the best way to do it, and trace-based testing makes that possible.  And then finally, on that similar vein, that means that developers take more ownership of their code. So this whole notion of not throwing it over the wall, which is what DevOps was meant to be but then got bastardized and turned into yet another layer. I would like to see that layer removed and streamlined a little bit more. I'm not saying take out DevOps as a concept. I'm saying take out the DevOps engineer and feel free to disagree with me. I'm sure that this might be a spicy take in some circles. ANA: [laughs] Spicy. ADRIANA: But that's how I feel about it. [laughs] ANA: No, I think that nails it. And I think that trace-based testing approach is very important on par to where we see the industry going. I know 2019, 2020, Andi Grabner and I were preaching test-based operations as we were advocating for Keptn. ADRIANA: Nice.  ANA: So I think it is a slow time coming that we are seeing this movement start happening. I remember in another time, in another wish list conversation I had with you, you wanted to see something more around organizations having SRE workflows that they share with one another. ADRIANA: Oh yeah, that's right, that's right. Yeah, yeah, because, again, honoring SRE principles there...and we actually...this can be a shameless plug for us. We spoke with Stephen Townsend of the Slight Reliability Podcast. We mentioned this as well where to honor truly the mission of the SRE of codifying all the things that are part of the mission.  There are so many instances where we find ourselves switching from job to job where we're like, oh, I've solved this problem already. But now I have to reinvent the wheel because that knowledge stayed at employer X. And wouldn't it be nice if we had a means of sharing some common workflows? Like, yes, the details are probably going to be a little bit different from employer to employer. But it would be nice if we had some framework, if you will, for that. And we see that a lot with CI/CD systems, more CI, less CD. Wouldn't it be nice to have an equivalent for SRE? So that's definitely top of my wish list. ANA: I'm definitely hoping that we make this happen with all the open-source tooling that we have that does all these things. But they're not doing it all together in a way that you can easily grab and take from organization to organization and that you get to then share those stories from organization A all the way to organization W. ADRIANA: Yeah, exactly. ANA: It's going to be a very, very exciting year. And we very much look forward to having all of our listeners here, the guests that we have that get to shed a little bit more of light into the world of SRE, of engineering, of being a human in the technology space, and such.  So with that, we would like to thank you all for joining in for Season 2 and very much look forward to having you all join us as we talk about SRE, observability, DevOps, on-call, and everything in between. Don't forget to subscribe and give us a shout-out on Twitter, Mastodon, LinkedIn, Instagram, or Tiktok. At this point, we're doing all the socials. [laughter]  ADRIANA: All the socials. [laughter]  ANA: Be sure to check out the show notes on oncallmemaybe.com for additional resources and to connect with us on social media. For On-Call Me Maybe, we're your hosts Ana Margarita Medina... ADRIANA: And Adriana Villela. Signing off with... ANA: Peace. ADRIANA: Love. ANA: And Code. [laughs]

About us: Adriana is a Sr. Developer Advocate at Lightstep, based out of Toronto, Canada, with over 20 years of experience in tech. She focuses on helping companies achieve reliability greatness by through Observability and Incident Response practices. Before Lightstep, she was a Sr. Manager at Tucows/Wavelo. During this time, she defined technical direction in the organization, running both a Platform Engineering team, and an Observability Practices team. Adriana has also worked at various large-scale enterprises, including Bank of Montreal (BMO), Ceridian, and Accenture. At BMO, she was responsible for defining and driving the bank's enterprise-wide DevOps practice, which impacted business and technology teams across multiple geographic locations across the globe. Adriana has a widely-read technical blog on Medium, which is known for its casual and approachable tone to complex technical topics, and its high level of technical detail. She is also a HashiCorp Ambassador. Find her on Mastodon at @adrianamvillela@hachyderm.io to talk all things tech. Ana Margarita is a Staff Developer Advocate at Lightstep and focuses on helping companies be more reliable by leveraging Observability and Incident Response practices. Before Lightstep, she was a Senior Chaos Engineer at Gremlin and helped companies avoid outages by running proactive chaos engineering experiments. She has also worked at various-sized companies including Google, Uber, SFEFCU, and Miami-based startups. Ana is an internationally recognized speaker and has presented at: AWS re:Invent, KubeCon, DockerCon, DevOpDays, AllDayDevOps, Write/Speak/Code, and many others. Catch her on Mastodon at @anamedina@hachyderm.io about traveling, diversity in tech, and mental health.  Find us on: On Call Me Maybe Podcast Twitter On Call Me Maybe Podcast LinkedIn Page Adriana’s Twitter Adriana’s Mastodon Adriana’s LinkedIn Adriana’s Instagram Ana’s Twitter Ana's Mastodon Ana’s LinkedIn Ana's Instagram Show Links: AWS re:Invent OpenTelemetry KubeCon Break Things on Purpose Podcast Transcript: ANA: Hey, y'all. Welcome to On-Call Me Maybe, the podcast about DevOps, SRE, observability principles, on-call, and everything in between. I am your host, Ana Margarita Medina, and with my awesome co-host... ADRIANA: Adriana Villela. ANA: Today, we're going to be recapping what this year has been for us, some of the stuff that we have been up to, what we're looking forward to in the next year, and talking a little bit more about tech culture and what we could be doing to make it better. We're so happy that you have been listening to us this season, and we look forward to having you join us next season. To kick it off, what's your drink of choice today, Adriana? ADRIANA: Today I've got water. Super boring. I should have something warmer, though, because it's actually really cold here in Toronto today. We've got snow, our first snowfall of the day. I mean, sorry, first snowfall of the year, I should say. ANA: Ooh, that's actually pretty exciting. Like, I'm not someone that likes cold environments, but snow is really pretty. I'm not going to turn down being in cute snow clothes with a little beanie. [laughs] ADRIANA: Yeah, totally. Exactly. As long as it's not super, super cold. ANA: Well, hopefully, you'll get some hot drink for the day to cozy up with the snow.  ADRIANA: Yeah, totally.  ANA: For me, today I'm also doing water, and I'm finishing up my morning drink, which is orange-infused yerba mate. I kind of sometimes switch between coffee to tea. So it's kind of nice. ADRIANA: Nice. Nice. Yeah, I think we have a similar drink in Brazil called mate. It must be the same thing. It's got to be the same thing. The name is similar enough. ANA: Like, at least the little bit of knowledge that I know about mate is that it comes from South America. And my best friend is Argentinian, and she actually just gets the actual leaves of mate. And you boil them, and you drink them in these little, leather metal containers that help keep it hot.  ADRIANA: Oh, yeah, yeah, yeah.  ANA: And you have the metal straw. So I've seen it more down there than in the cultures that I come from. ADRIANA: That's cool. Now it makes me want to go to the nearby Brazilian supermarket and pick one up. [laughs] ANA: You should do it. ADRIANA: For nostalgia. I should; I should. Next time I'll come equipped with some mate. We can have a mate-mate off. [laughter] ANA: I'll make the OG Argentinian mate for that day. ADRIANA: Nice. [laughs] ANA: Is there any drink that in Brazil y'all drink for holidays or that you and your family really like having as a tradition? ADRIANA: No. I mean, I think I dishonor my culture because I don't like coffee. [laughs] My dad's really into coffee, but my mom never was into coffee. I'm much more of a tea drinker. Or I'd say my go-to holiday drink is a Starbucks hot chocolate, and I usually do it with coconut milk, which gives it a really nice silky texture.  ANA: Ooh, I might have to try that. I just had their hot chocolate maybe two weeks ago. And I forgot that I actually really enjoy getting a hot chocolate from Starbucks. Like, it's a different experience than just making it a home. ADRIANA: Yeah, it's like a warm hug. [laughter] ANA: Yes, that definitely sums it up. I think for me, like, I just blanked out. But in my culture, we have different things that we do for the holidays. And in Costa Rica, we have rompope, which is our traditional eggnog that we drink during the holidays, and it's literally my favorite thing to drink. It's just not really available out in California. I think in Miami, it was easier for me to find it. So over the last seven years, what I have picked up as my holiday tradition is making coquito which is a Puerto Rican drink. ADRIANA: Ooh. ANA: Which is like coconut milk, condensed milk, evaporated milk, cinnamon spices. The culture of Puerto Rico is that you make it with rum. So you can do it with Brugal, and you serve it over ice, and it's extremely strong. So it's more of like their holiday party drink. So I always make it non-alcoholic, and then I'll pour Flor de Caña Nicaraguan rum and have it with it. And it's just like, cozy up. It's the holiday season, like, start celebrating from Thanksgiving to the end of January. ADRIANA: Oh my God, that sounds amazing. [laughter] I will have to try that at some point. You had me at condensed milk, so... ANA: Yes. I mean, that's what it is. It's that sweetness. It's that sweetness and boozy. It's also a culture that you make it for your friends and family, and that's your gift for the holiday season. ADRIANA: Aww. ANA: So, one of my past jobs with a credit union in Miami, I got gifted it in like one of the holiday seasons, and it was just really thoughtful of like another family is inviting you to celebrate holidays with them.  ADRIANA: Oh, that's so sweet. I love it. So, we've been podcasting for a while, I guess, now, right? So I guess this is our year in review [laughs], where we get to reminisce on some of the highlights of the year. You know, it's a good time for reflection. So keeping in with that end-of-year theme, like, what experiences are you grateful for? And who are some of the people that shaped who you are today? ANA: Yeah, I mean, I think things that I'm grateful for have varied a lot year by year. I think this year I'm really grateful for the close group of people that I have around me; it has been co-workers, husband, open-source community. It has been my close friends and just uplifting. I remember the last winter season; I had a lot harder time mental health-wise. So kind of trying to remind myself of, like, oh no, like, leverage your community to not struggle this much this winter as we already have the sun going down at 4:30 p.m., which just makes my heart cry. [laughs] ADRIANA: So depressing. I hate it.  ANA: [laughs] ADRIANA: The only thing making it less crappy today is the snow.  [laughter] ANA: True. Send some snow over to California, please, but not too much. [laughter]  ADRIANA: Yeah, just a little, tiny bit.  ANA: And then, to answer your second part of the question, I think there are a lot of people that I'm grateful for for making me who I am today. I think my dad instilled in me a lot of values of being community-driven, giving back to folks, and constantly just learning and staying curious, like, an entrepreneurial mindset.  And then there are two mentors that I always go back to as folks that made such a huge impact in my life growing up that I wouldn't be in the career that I have today. And one of them is my sixth-grade technology teacher, Mr. Rios, who taught me just computers and the web are a really cool thing. And he published my HTML websites on his website.  ADRIANA: Aww. ANA: And that gave me confidence of technology is cool; go code. And then, I had a manager for an internship in my ninth-grade summer of high school named Soleil. She was just an amazing woman of color leader that was fierce in business and really tried to explain to these high school students, like, you have to present yourself the way you want people to respect you and also think about your career really early, and those decisions are going to make an impact. So if it wasn't for them, I wouldn't be having a successful career in technology. So I think this holiday season, I'm always like, hmm, thank you. Thank you, all. ADRIANA: Aww, that's awesome. That's really sweet. ANA: What about for you? Who are you grateful for that has made you who you are today? ADRIANA: I'd have to say top one is my mom, and I lost her earlier this year to cancer. And I had to actually rush home early from KubeCon to be with her for her final hours, and that was really rough. But I'm also so grateful that all my family was there. My dad, my sister, my daughter, and I were there for her final moments. And every day, I think back to the lessons that she instilled in me, which were she always wanted my sister and me to be fiercely independent career women, and she got her wish.  I mean, for her, the most important thing was no matter what, just make sure that you set yourself up that whatever life throws at you, you're able to come out maybe with a few scratches, but you still come out okay in the end, and I'm so super grateful for that. And she's always taught me to be super driven. And I think we both, [laughs] you know, I'm pretty sure she was undiagnosed ADHD. She was like a little rocket ship. I mean, we'd take 10K walks regularly. I think I definitely get that from her. So I'm very grateful for that.  And I would say my dad and my sister have been so supportive in the last little while, like, after losing my mom and gotten closer over that, which has been super nice. My dad is actually the one who got me into tech. [laughs] He's a software architect. I mean, last week, he's like, "Yeah, I'm learning this new language called Zig, and it's supposed to be like an easier version of Rust." [laughter] I'm like, holy crap, man, you eat computer languages for breakfast. [laughter]  So yeah, he's always on top of the techy things. I got into DevOps because of him. [laughs] When I was bitching one time after a really bad release, and all the things that went wrong did, and he's like, "You should look into this DevOps thing."  ANA: [laughs] ADRIANA: I'm like, ooh, cool, and that got me hooked, so eternally grateful for that. And my husband has been ridiculously supportive of my career. And he's been at the same company for, like, 25, 26 years. And I'm the one who's had career changes and job changes over the last 20 years of our marriage, and he's super chill about it. [laughter] Even though at one point I'm like, "I'm quitting tech, and I'm going to become a professional photographer," and he's like, "No problem." [laughs] ANA: Aww. ADRIANA: So that was cool. And my daughter, Hannah, like, honestly, she's the sweetest person. And she's always there to cheer me up whenever I feel crappy, so I super appreciate that. And I've had a couple of really awesome mentors, too; my friend, Bernard, who unfortunately suffered a stroke a couple of years ago and is still in hospital unresponsive, but I always think of him because he's the one who got me interested in observability. And he's one of those people who could see...I guess he saw what my mom saw in me. He's like, "Oh, you're really good at what you do. And you should never forget that."  It's so nice to have someone who believes so fiercely in you and can see past, you know, you're your own worst enemy. And I had so many mental blocks, like, doubting my skills. And so he continues to inspire me. And every day, I think of him and every day where I am right now, especially in this career as a DevRel, in observability I'm grateful to him for that.  And another mentor I can think of is a manager that I had many, many moons ago named Lawrence when I worked at Bank of Montreal. He was just like the most supportive manager, and I feel like he's the gold standard of managers because he was easy to talk to. I could play pranks on him.  ANA: [laughs] ADRIANA: I got him a bullshit button as a joke one time, [laughter] and he took it super well. And I also got him a giant calculator as a prank, [laughs] and he took it all in stride. And I think being able to have that kind of a chill relationship with the manager while they're still interested in helping you grow and mentor you I think that's priceless, so yeah. ANA: I mean, most definitely. It's like, I see you as a human, and I'm going to help you flourish in every single personal way that I can versus only in the box of tech or in the way that you bring value to an organization.  I was going to mention that I'm still very sorry about the loss of your mom earlier this year. And I'm glad that you've been able to spend time with your family to be together and help build each other up because there's a little hole in your life in the holiday season. I know it gets hard. And I know you, and I have talked about it of what it's like to lose parents and continue on. The best we can do is just make them proud. ADRIANA: Exactly. Exactly. Honor them by honoring their teachings.  ANA: Yeah, most definitely.  ADRIANA: I also want to say a big shout-out to our DevRel family because, honestly, y'all have been super supportive throughout all this. I had to rush home on October 27th, and everyone's like, "No problem, do what you need to do, like, no sweat." And everyone's just been, like, I don't know, you guys sent me some lovely flowers and just some lovely messages. And it was super nice. So really appreciate it. Y'all are awesome. ANA: We're humans first. That's all that matters. We just get to work on really cool technology and get to build communities around it. ADRIANA: I'm so grateful to have the opportunity in the last couple of jobs to work at places that are less robotic about the corporate-y stuff and really value the humanity of the employees. Like, I don't feel like a number. I feel like the work I do matters, and I think that's so important. ANA: I've worked at places where it said that humans come first, and then things change. And I think a lot of it also varies team by team, manager by manager. Things can change really quick, which really sucks. But I try to be an advocate for that even when the situation around me might not be like that. It's like, nope, I'm putting my foot down. We are humans.  There are limits to our capacity, to the things we can do. We can't just be running around trying to do 50 tasks all at once or working 60-hour work weeks continuously. I understand that it might be needed once in a release or something like that to catch up. But there are limits, and life is not just all about tech and our jobs. ADRIANA: Yeah, yeah, I totally agree. Very well said. ANA: [laughs] I think one of the things that talking about gratefulness, and talking about end of the year, and looking back into what we have done, I think one of the questions that I ponder about as we have this conversation is how is it that we can build more of a thank you culture in technology? Like, working within the Kubernetes community, some folks try to constantly be giving like, hey, thank you to this person that's helping out on this enhancement that we're working on or on this PR. But I think in the wider technology space, we don't always give honor to those that came before us and helped set a certain guideline or a certain movement. ADRIANA: Yeah, that's true. And I think remembering where it all came from, even Kubernetes community or OpenTelemetry community started from somewhere. And those pioneers of the early days, like, that was some serious hard work that they put in. And things might seem like fairly well-oiled machines, I mean, nothing's perfect, but compared to the start, I'm sure it's a lot more organized than it was at the start. And we got to give kudos to the people who cared enough to build up those communities and to the people who continue to care to continue to support these communities.  Because, I mean, open source wouldn't be where it is if there wasn't a sense of empathy and understanding and not making people feel crappy or scared about opening PRs, even though I'm still scared every time I open a PR. [laughter] But knowing that there is that psychological safety in the community that I'm working in where there's no judgment, there's gentle nudges of like, oh, perhaps you should look into doing XYZ instead of ABC, so then it makes you feel it's a lot less scary. ANA: Yeah. And I think with that too is, as you mentioned, I think you have to be really intentional about it. It just doesn't happen overnight, and it doesn't stand on its own. Like, if a person is pioneering it, once they go away, they don't have that in that community. So how can you replicate it and make sure that it's scaling, like, the empathetic aspect of it, the human side of stuff, and also constantly build each other up and support one another? ADRIANA: Yeah, I totally agree. Yeah, I think we need to give out more kudos in our tech communities. That's one thing that I do appreciate in our internal slack, like, we've got a kudos channel or whatever, whatever the channel is called where people give each other kudos for even things that you're like, oh, you know, I've gotten kudos for participating in something. But it's like, yeah, I was there helping out with whatever. And it's kind of nice to get some recognition, and it feels nice. And it feels nice to give it too, because it's like, hey, I appreciate the work that you're doing. ANA: I mean, it's building trust too. Like, you helped out; I help out. We're in this together. We're building something bigger. And I think that recognition culture also gives people that sense of purpose and just fills that need. So I also agree. Like, in my last organization, we called it tacos instead.  ADRIANA: Nice. ANA: So you got tacos for everything that you...for people giving you, like, thank you for helping out on this. And they could be really random to you closed a customer. [laughs] So it's a lot of fun ways to think about it. But yeah, I would love to hear also some of our listeners' thoughts on that, like, what can we all be doing to build more of that culture? ADRIANA: Yeah, absolutely. Please feel free to reach out to us in our various social media accounts with your feedback. We truly, truly want to hear back. ANA: With it being the holiday season, are you going to be taking some time off to recharge? ADRIANA: Yeah, I always take a couple of weeks off during winter break to line up with my daughter's winter break at school. So it's nice because, typically, we don't go anywhere during December. It's just a hunker down at home and chill kind of holiday, which is super nice. And then hanging out with family and making Christmas dinner and stuff, and we all cook together, which is fun. So it's not like any one person's burden, so I definitely appreciate that.  And maybe I'll venture out this year and go ice skating. I'm not the most coordinated when it comes to winter sports, but I still make an effort. [laughs] I can ice skate and "ski" in air quotes. I do enough of a decent job that I'm not falling on my ass on a regular basis. So I'm hoping to get out there this [laughs] winter, yeah.  ANA: Nice. ADRIANA: Crush it. ANA: Hopefully, that means some photos will be coming on social media of you, [laughs] you on winter sports. ADRIANA: Oh yeah. [laughs] There's a video of me skiing a couple of years ago (I hadn't skied in like 15 years.) where I'm like seriously a grandma on the bunny slopes making these wide-ass turns because I'm like, I haven't skied in a really long time, and I'm not ready to commit to a steep, steep hill. [laughter] ANA: I still have yet to do winter sports, so I do not relate to any of that. ADRIANA: [laughs] ANA: But I can see the visual in my head, and I would love to watch this video. [laughs] ADRIANA: Okay. Yeah, I'll dig it up. I got to dig it up. [laughs] ANA: And for me, I'm also going to be doing winter break, taking time off. I haven't figured out plans, I think, for this year. A portion of it, we'll try to stay put, try to explore outdoors. I think I just need to recharge with being out in the woods, by the ocean, even though it's going to be chilly. Being away from technology, I think, is what my goal is for whatever length of winter break that I take, like, just disconnect as much as I can, try to be with myself, introspect, like, think about what I want to bring in the next year. ADRIANA: Nice. Yeah, yeah, I do actually take my winter break as a technology break as well. So I totally agree. It's a nice time to take that break. ANA: Yeah. And, I mean, I think for the job that we do that we're developer relations, developer advocacy, that we're constantly in front of people. [laughs] And being in front of people is amazing, like, I love it, but it requires a lot of energy. And then we're on social media promoting stuff, talking to folks, doing podcasts, and things like that. For me, that December time after our re:Invent conference starts being like, slow down. And it's like, I got to take care of me in order to survive the next year. So let's hermit crab now and get under my weighted blanket, turn on the heater, turn on my candle. ADRIANA: Yeah, I totally agree, yeah. And it's going to be a busy couple of weeks for you because you're just coming off the heels of KubeCon, and now you're preparing for re:Invent. And I guess, well, there's Thanksgiving next week, and then re:Invent right after Thanksgiving, right? So... ANA: Yes. Yes, yes. It's definitely crunch time right now. I'll be speaking at re:Invent, and that will happen right before this episode launches. So yeah, giving a talk on using the Well-Architected Framework to build for liability, which should be exciting, and I'm very excited to put out that new content. I've worked with the Well-Architected framework for a while but never have built a special talk around it. It's been exciting to work on it. And it'll be nice to be closing all my tabs and logging off once that talk is done. ADRIANA: [laughs] ANA: We'll have some other projects going on, but they're not as big. That's kind of like the big deliverable. ADRIANA: That's awesome. That's going to be so satisfying. And hopefully, by the time this episode airs, we'll have a link to your talk that we can post in the show notes as well.  ANA: Yeah, if not, I'll publish my slides. It'll be very exciting to see how large AWS re:Invent is this year. I actually still haven't gone, as we're still in this pandemic. My last time there, I believe, was in 2019. So it'll be exciting to see.  ADRIANA: Is it always in the same place?  ANA: Yes. It's sadly always Vegas.  ADRIANA: [laughs] ANA: And I don't know what type of contract they have or why they do this. Or they do it because it's going to attract tech people, and sadly, that's a lot of male dudes. So I don't know if that has anything to do with it.  ADRIANA: [laughs] ANA: But I really wish that we would stop hosting AWS re:Invent in Las Vegas. Thank you very much for coming to my TED Talk. [laughter] ADRIANA: There's your PSA. ANA: Yeah, the amount of things that happen there that make me cringe it's a lot. So that's a whole other podcast episode that probably is not a podcast episode but should just be a Mastodon, Twitter, LinkedIn post.  [laughter] ADRIANA: Stay tuned, folks. Some juicy content possibly heading your way. [laughter] ANA: Possibly, we will see. ADRIANA: Possibly, if you get around to it. [laughs]  ANA: Yes, exactly.  ADRIANA: If it's still fresh in your mind at the time. [laughs]  ANA: Yeah. And, I mean, it's also we have so many little tasks to take care of in DevRel that it's easy to put that off as non-priority type of content.  ADRIANA: Yeah, that's true. ANA: So I think that's in the bucket that I have it on. Which actually brings me to my next question, Adriana, now you've been in DevRel for half a year. How has that experience been? ADRIANA: I have really enjoyed it, actually. I think for me, it's like a dream career because, in a way, I've been doing that to a certain extent in previous roles. Like, when I worked at Bank of Montreal, I ran a DevOps practices team, and we came up with practices and just evangelism around DevOps. And, I mean, that's very DevRel-y really kind of work. And there was a lot of blogging involved, as you know, one of my favorite things to do.  And then, even when I was working at Tucows in the observability practices team, again, there was a lot of evangelism trying to get people sold on observability, coming up with those practices, writing about those practices, lots of presentations on the benefits of observability.  So I feel like it's like the perfect fit for my personality. So I've been having a blast so far. I feel like the job keeps me on my toes. I'm always learning cool things. For me, that's the most important thing about our job. Like, as long as I'm excited to come into work, as long as I'm learning cool things on a regular basis, sign me up. When you get the bored version of me, that's when you got to worry. When the apathy starts to set in, be concerned.  But right now, I'm super stoked to be where I am. I'm loving the job. It's definitely been a learning curve. And thank you for guiding me along this DevRel journey because showing me some of the ins and outs has been super helpful, like even getting me on the DevRel Collective to connect with other DevRels. And even like when we were at KubeCon, getting to just show me the ropes of navigating KubeCon, that was super awesome. So yeah, I appreciate it. ANA: I'm glad you've been enjoying it. It's a fun role, especially in the space of SRE and DevOps, because it is very much culture-oriented and a lot of digital transformation. And when you come with the experiences of putting humans and people first, I think we need to build those cultures a lot more in the tech industry. So that perspective always helps.  But it's also a technology space that doesn't have that much content to help you learn. It's like you're expected to know a lot of this information because you have been working in tech for 20 years. So I think this kind of role allows for us to bridge that gap and allow for new talent to come in; at least that's been my experience. It's made me feel like it has more of a purpose. ADRIANA: Yeah, totally. I think you alluded to it earlier; it puts a human touch on tech. Like, hey, come learn with us. We're here to guide you. It doesn't have to be scary. Yay, come join us. And I've actually really enjoyed the various DevRels that I've interacted with, either on social media or even getting to meet folks in real life.  It's so nice because I do find at least the DevRels that I've interacted with it's a very open and welcoming and diverse community. Generally, folks are super chill. There isn't that stiff upper lip vibe that you get in the enterprisey, the corporate-y world, which I super appreciate because I am not that kind of girl.  ANA: [chuckles] ADRIANA: So it's nice and refreshing to see people being themselves, like, hey, you've got funky hair, awesome.  ANA: [chuckles] ADRIANA: You dress in funky clothes, awesome. Love it, be yourself. And everyone accepts you for who you are, and I think that's the best thing ever. ANA: Yeah, I do really like that we try to help set a new standard for the technology industry of bring your whole self to the space. We're going to be happy to welcome you and open doors for you in any way that we can since a lot of our job is building connections and relationships. And we bring good into the technology space. [laughs]  Of course, I'm very biased in that sense but being able to really listen to developers and build around the use cases that they're struggling with. Or have someone actually listen to their feedback and the problems that they're having in scaling their services, or even just hearing about burnout and giving our thoughts on ways that the organization can improve their observability incident response practices along the way. ADRIANA: Yeah, yeah. And I think that's a really key thing is like, you have to be willing to listen as a DevRel. Like, you can't just be on your soapbox in your ivory tower preaching this is the way it should be, folks.  ANA: [laughs] ADRIANA: You have to interact with the folks on the ground who are doing the thing that you're talking about. And if you're not willing to listen to that feedback, then what the hell are we here for? ANA: Yeah. And, I mean, and if you can, do the work too. So if you're able to sit in your organization with an engineering team and do work, work on product launches, to build demo environments, build pre-staging type of environments to really be able to keep your technology shops in touch, like, [laughs] I can't think of words. But stay in the loop with technology trends, I think, is what I'm trying to say and be able to relate to the community.  But also, think about what's coming next in the space. Like, what are the problems that developers are starting to come across when they've been working with this type of technology for a really long time? Which I think is what we've seen a lot in the evolution of DevOps tooling and SRE tooling in the last few years. ADRIANA: Yeah, yeah, absolutely. So, on a similar topic, I think you've touched upon some aspects, but I'm a noob to DevRel. You have been in DevRel for a while. But what are your impressions? Because I mean, yes, you're not new to DevRel, but you're newer, I would say to observability. So how has it been? And we started at Lightstep a week apart from each other. So we're like onboarding buddies, work sisters. [laughter]  ANA: Yes. Having you join a week earlier definitely helped because I came in with a lot of questions, and having a resource on my team that was also probably a little lost as I was was really cool. [laughter] ADRIANA: And like-minded too. ANA: Yes. Yeah, there's definitely, like, we share a big portion of our brains together.  ADRIANA: [laughs] ANA: It's kind of been interesting to work with you over the last six months. It's been fun. But it's nice to have that safe space of asking questions. Because I have worked as a site reliability engineer. I've done software engineering. I've done the DevRel for a few years. But when it came to observability, like, I always understood it. I would work with customers that did observability work, but I never was that person that would try to instrument their code or run any auto-instrumentation library.  So onboarding to Lightstep on observability was me trying to nail down the concepts of observability from spans, traces, metrics, logs, events, trying to just understand how they all differed and have them make more sense in my head and then go back to a mindset that I had early on in site reliability engineering where you really do need to understand your system in order to build reliability.  That allowed for me to evolve a few of my talks a little bit more about that of, like, in order for you to actually do the work in reliability, you have to start with looking at where things are at right now. How can you observe them more? How can you be measuring properly or even standardize them with service-level objectives and such? So it's been pretty interesting.  I also just got to actually do more work in observability by working with OpenTelemetry, which was my first time touching OpenTelemetry. I remember maybe month two, month three spinning it up locally with Docker, and having my application, and putting in my traces into Lightstep. And it was like, cool, I got things working, but I hadn't really tried to dive down into code.  So OpenTelemetry this past month, or two months ago [laughs] by the time this episode launches, has released an OpenTelemetry demo, which is a vendor-neutral instrumented application that comes in various languages. It's a microservice application that the services are written in multiple languages. So some of the work that Adriana and I did for KubeCon was introducing metrics into the application. So I took a first stab at it, and it was opening up like this Python application. And it was like, we have to implement [laughs] metrics into this. ADRIANA: And we had no idea what we were doing. [laughter]  ANA: So it was, like, there's no documentation because things are being built out on the open-source side of things. I've never really been a Python developer. I can do it for scripting. I can do it for tutorials for, like put things together PoC-wise, but in terms of making things work with no documentation, it was really hard. So it was really fun to pair with Adriana in terms of asking her questions of, like, wait, you mean, OTEL has like a logger that I could just co-log there and make sure that we're setting up the instrumentation properly? ADRIANA: [laughs] ANA: Like, it was interesting to just realize how much easier the development process was if you leverage observability. ADRIANA: Yeah, yeah. And that was a super fun project for us to work on. And it was cool, too, that we got to introduce our work as part of KubeCon at the Lightstep booth. There were some demos going on around that. I'm super proud of the work that we did around that because [laughs] there was like blood, sweat, and tears put into that. I thought the Python stuff was hard, but then the Terraform issues we ran into afterwards, I was like, oh my God, Terraform, why do you hate me so much? But we got past it. We made it work. [laughter] So I'm super happy with that.  It falls under the category of getting to learn cool stuff as part of the job. So I'm very, very grateful for those experiences. I get such a high from being able to solve problems like that, even if [laughs] to other people; it's like small potatoes. But for me, it's like, I figured it out, and it was awesome. [laughter] ANA: Yes. And, I mean, I think it was also we also had that nice wrap-up of we had been working together for various months and had been doing calls three, four times a week. And then we finally got to meet in person at KubeCon in Detroit, which I think it's also a highlight for On-Call Me Maybe.   ADRIANA: Yes. ANA: And for just in general, us [laughs] of like, oh cool, one, we are two short Latinas. ADRIANA: Yes, yes.  [laughter] ANA: And two, it's cool to meet people you work with. [laughter] ADRIANA: And your height. [laughter] ANA: That was definitely a nice little highlight. ADRIANA: Yeah, it was fun and meeting the rest of the team in person as well. Getting to meet Austin and Ted in person was super fun. It definitely, I think, helped us bond more as a team just from that relatively short interaction in person. But it was super awesome. ANA: And it was also really cool because we got to meet listeners of On-Call Me Maybe.  ADRIANA: Yes. ANA: I know I had two, three people come up to me, and, like, we're talking at an event. I'm not wearing a nametag or anything. And then they just kind of look at me, and they're like, "I recognize your voice." And I was just like, "Wait, what?" ADRIANA: Oh my God, so cool. ANA: And then they're like, "On-Call Me maybe." And I was like, "Yes, I'm one of your hosts for On-Call Me Maybe." And this is my second podcast. I did Break Things On Purpose podcast a few years ago. But it was really cool to finally get acknowledgment for a podcast episode. Like, I'm still not comfortable with this type of medium but to know that our audience is loving it is kind of nice. So if you ever see Adriana and I anywhere, or you're liking the content, we would love to hear y'all's thoughts on it. ADRIANA: Yeah, and come say hello on social media or in person if we happen to be at an in-person event that you're at. We love connecting with people. I also got a similar thing where they're like, "Oh, I recognize your voice," because I guess they didn't know what I look like because my avatar on Twitter is like a cartoon, and so...[laughs] but they recognize my voice, which I thought was super cool. ANA: Yeah, it's kind of nice. And, I mean, it's been really nice to be able to grow the On-Call Me Maybe community with all the listeners too. And we're really excited for some of the speakers that we have coming next year, folks that are working in open source, folks that are working with eBPF. What else do we have in store for us? ADRIANA: I think we've even got some folks like, you know, one of our guests works in agile, so we'll get some hot takes in agile. [laughter] So that'll be exciting. And also to put it out there, for anyone who has something cool to say about the space that we work in, observability, SRE, DevOps, and anything in between, reach out to us if you're interested in being a guest on the show because we would love to hear from you. We would love to talk to you.  It's so much fun being able to meet different people and interact with different people from different areas of tech. It's been so much fun. And I'm definitely super grateful for having had this experience. I really enjoyed podcasting. I think for me, I was a little nervous because this is my first podcast. So I was definitely a little bit nervous at the start thinking, oh my God, am I going to be stiff on air?  ANA: [laughs]  ADRIANA: And I hate the sound of my voice. [laughter] But it's been lots of fun. It's been super chill conversations. It's like going out for coffee with friends is what this podcast feels like. So it's been awesome. ANA: That's awesome. I feel very similar thoughts on that. Definitely, I was a little skeptical on jumping on another podcast, but it's been great to have an awesome co-host and to talk to so many amazing guests that we've been able to this season.  With that, we're going to be closing off. We would like to thank you all very much for joining us for today's episode, recapping what the year has been for us, letting us share a little bit more about the reasons we're grateful for the work that we do and the communities that we work with.  Don't forget to subscribe and give us a shout-out on social media. You can find us on oncallmemaybe on various platforms. And be sure to check out the show notes on oncallmemaybe.com for additional resources and to connect with us on social media. For on Call Me Maybe, we're your hosts, Ana Margarita Medina... ADRIANA: And Adriana Villela. ANA: Signing off with peace... ADRIANA: Love and... ALL: Code.

About the guests: Austin Parker is the Head of Developer Relations at Lightstep and has been creating problems with computers for most of his life. He’s a maintainer of the OpenTelemetry project, the host of several podcasts, the organizer of Deserted Island DevOps, an infrequent Twitch streamer, a conference speaker, and more. When he’s not working, you can find him posting on Twitter, cooking, and parenting. His most recent book is Distributed Tracing in Practice, published by O’Reilly Media. Ted Young is the Director of Open Source Development at Lightstep and one of the core maintainers of the OpenTracing project. Ted has spent the last 15 years building distributed systems in a variety of environments: computer animation, national elections, and elastic computing platforms.  Find our guest on: Austin’s Twitter Austin’s Mastodon Austin’s LinkedIn Austin’s Website Ted’s Twitter Ted’s LinkedIn Find us on: On Call Me Maybe Podcast Twitter On Call Me Maybe Podcast LinkedIn Page Adriana’s Twitter Adriana’s Mastodon Adriana’s LinkedIn Adriana’s Instagram Ana’s Twitter Ana's Mastodon Ana’s LinkedIn Ana's Instagram Show Links: Lightstep 360.org Bill McKibben The Sunrise Movement OpenTracing OpenCensus OpenTelemetry KubeCon gRPC BASIC Programming Language DockerCon CNCF CoreOS Liz Fong-Jones OTLP (OpenTelemetry Protocol) QPS (Queries per Second) New Era Caps Denise Yu Hydroflask Bottles Star Trek: Voyager Lava Lamp Additional Links: Distributed Tracing in Practice (O’Reilly Media) The Future of Observability with OpenTelemetry (O’Reilly Report) [YouTube] Myths and Historical Accidents: OpenTelemetry and the Future of Observability Part 1 [YouTube] Data by Design: OpenTelemetry and the Future of Observability Part 2 [YouTube] What OTel is and isn't: OpenTelemetry and the Future of Observability Part 3 Transcript: ADRIANA: Hey, y'all. Welcome to On-Call Me Maybe, the podcast about DevOps, SRE, observability principles, on-call, and everything in between. I am your host, Adriana Villela, with my awesome co-host... ANA: Ana Margarita Medina. ADRIANA: And today, we're talking to the whole DevRel team at Lightstep, woo-hoo. ANA: Woo! ADRIANA: And it's taken us how long to get us all together on this? I think this is our third attempt. ANA: Yeah, it's actually third attempt. It's taken five months in the making. ADRIANA: [laughs] ANA: But it's here. We're finally getting this episode out the door to our listeners. ADRIANA: Yay. So we start with...our first question is always, what are y'all drinking? AUSTIN: Red Bull, sugar-free  TED: Coffee.  AUSTIN: I had an espresso earlier. ANA: Y'all are kind of crazy to still be doing caffeine at this time, like, East Coast 4:00 p.m., Pacific time 1:00 p.m. I have a strict cut-off of no more caffeine at 12:00. I just finished drinking my yerba mate, 160 milligrams. AUSTIN: Yeah, I drink caffeine. I don't know; I don't really have a ton of problems with caffeine. So earlier this year, when we did Deserted Island DevOps in Michigan, I was like, okay, as the organizer of this, I can finally make unreasonable craft services requests. And so I was like, we got to have sugar-free Red Bulls because I like energy drinks. I like sugar-free Red Bull.  So as it was requested, there was a, you know, they would fill it up. And that was actually maybe a mistake because I think I drank like three or four of them, and I was too nervous to eat. So I think I just kept drinking sugar-free Red Bulls. And about noon on the first day, I felt like I was vibrating on my skin.  ADRIANA: Oh my God.  AUSTIN: So I did find my limit of sugar-free Red Bull, and it's about four in two hours. ANA: [laughs] ADRIANA: Yikes. When I was in university, I used to have a can of Coke usually for a 9:00 or 10:00 a.m. class because I don't drink coffee. [laughs] I remember experiencing the vibrating in my skin sensation back then, and then I'm like, yeah, maybe I shouldn't do this anymore. TED: Caffeine has no discernible effect on me anymore, which in and of itself is probably bad. [laughter] ANA: It might mean something. AUSTIN: It's a sign. It's a signal. [laughter] It's some kind of signal, at least. [laughter] TED: The adrenal glands are just like, we give up. We give up. AUSTIN: So we've stopped trying. This guy is going to do whatever. [laughter] ANA: I've switched over to water for the rest of the day. So, what about you, Adriana? ADRIANA: I've got water. I do plan on having a lovely mug of green tea later, though. Caffeine doesn't; at least green tea doesn't affect me if I have it late in the day. I'll have homemade bubble tea at like 9:00 o'clock at night, and I'll be fine, so yay. TED: With your own tapioca boba? ADRIANA: Yeah, yeah. TED: Whaaat? ADRIANA: Yeah, I get it on Amazon, yeah. And I can control the amount because I like it, but I find it too filling. So I'll have like just a few.  ANA: Two pearls.  ADRIANA: Yeah. [laughs] Sometimes five if I'm feeling super adventurous. But yeah, it boils in like one to two minutes, and then it's done. So that's my plug for homemade bubble tea.  TED: Whoa. ADRIANA: Mind-blown. [laughs] AUSTIN: I've not had boba in so long. I miss it. I used to get them when I came up to San Francisco when I would travel, back before the pandemic and everything. I would always really get into some boba.  ANA: That was the case when I worked at the San Francisco Uber office. We would always take walks around the block, and we would end up at Boba Guys or Asha Tea House. And it was just like the perfect 45-minute break, nerd out, get some sun, get boba. Sometimes it was a work expense because it was a work meeting, and I was like, this is nice tech privilege. ADRIANA: Yes, yes. I would totally do that, too, at one of my old jobs. It was like the daily bubble tea round. [laughs] And I used to work in Downtown Toronto, so there's like tons of places now. So never wanting for bubble tea. [laughter] TED: They always dunk way too much sugar into it for my liking.  ADRIANA: Yes.   TED: So the idea of homemade boba sounds really enticing. ADRIANA: Yeah, that's why I like it too because I find they do put too much sugar, and I can control the amount of sweetness because I can't even take the amount of sweetness that Starbucks puts in its hot chocolates, like, it's sugar overload for me. ANA: When I request bubble tea, I'm always like the 75%-80% sweetness person, so I'm the complete opposite clearly. ADRIANA: [laughs] I mean, you do enjoy condensed milk in your drinks, too, right? ANA: And I cannot do sugar-free Red Bulls, like, no. Sorry, y'all. [laughter] AUSTIN: I mean, it's fine. ANA: I mean, you're caring about your health which is better than what I do, so I think -- AUSTIN: I mean, I like sugar in things, but I just don't like it in drinks these days. I don't know; I've been trying to reduce sugar intake in general. TED: I mean, it's not like regular Red Bull tastes good, so you might as well get the sugar-free, right? ADRIANA: [laughs] TED: Like, it's not like you're losing something.  AUSTIN: Well, that's true. I mean, there are not a lot of energy drinks that taste good. Is this just like an energy drink review podcast now? [laughter] ADRIANA: I guess so. AUSTIN: All right. ADRIANA: Apparently, apparently. [laughter] ANA: I still think Green Monster tasted really good growing up. That was definitely a go-to. AUSTIN: Hmmm. See, Monster Zero, I think, tastes better than the regular ones. And then, for a while, I went for the Bang energy drinks. Those are a lot.  ANA: I just get turned off by the marketing.  AUSTIN: Well, the marketing is also bad. I feel like we need a new kind of energy drink revolution or energy drink model or concept, like something -- ANA: There is actually something new that I just discovered. And I discovered it because somehow I was going through Twitter, and I found a founder who follows me on Twitter who's also Costa Rican, lives in San Francisco, and is building a company on making coffee without coffee beans.  ADRIANA: That's cool.  AUSTIN: Interesting. ANA: I don't know the name of it.  TED: What does it mean?  ADRIANA: [laughs] ANA: It's using, like, cocoa nodes. I don't know how they're not processing the cocoa beans, but you're still extracting caffeine. And you still get the same flavor profiles that a lot of coffee lovers are able to get out of it. At least, that's what I gathered from reading her tweets or company's tweets.  AUSTIN: Is it like a sustainability thing or?  ANA: Yes, yes, yes. It's a sustainability angle of it, too, where it's like with the way that climate change is going, we're not going to be able to keep up with the coffee demands, so we need to look for alternative routes. AUSTIN: I like people when they try to head off, or they look at the dystopia, and then they just kind of like say, "You know what? We're going to just jump one step ahead of that. ADRIANA: [laughs] AUSTIN: We aren't at the collapse quite yet but let's go ahead and see how much VC cash we can wring out of monetizing the rot."  ANA: I mean, I'm super biased as being from Costa Rica, but they're one of the biggest exporters who are known for their beans that I'm like, to be a founder...that it's like, as a country, we stand for taking care of our environment and sustainability always being part of the way that you grow up, like, to then carry that on in Silicon Valley, I was just clapping hands, just excited. [laughs] AUSTIN: I know it's a cool idea. Don't get me wrong on this. ANA: Well, I will take us to our technology podcast. ADRIANA: Technology. [laughs] Yes, let's get into the technology meat of it. So I guess, for starters, Ted was my co-host on one of the On-Call Me Maybe episodes where we had Luiz Aoqui talking about Instrumenting Nomad with OpenTelemetry. Let's use this as an opportunity to get to know Ted and Austin better. AUSTIN: I nominate Ted to go first.  ADRIANA: Awesome, awesome. [laughs] I feel like when we met in person at KubeCon, you striked me as way more interesting than me. [laughs] So I feel like you've got really cool stories to tell, like how you got into tech and how you got into OpenTelemetry. [laughs] TED: I have definitely lived a very non-linear life, which leads to, like [laughter], oh, there's the one time I was racing an electric car in Denmark kind of thing or whatever. Tech, I initially moved to San Francisco to be an animator. So I guess I got started in the animation industry but in 3D animation. And I did have a computer science degree, so it was very techy. Besides animation, I was really interested in what's called rigging, which is where you set up the armature, like all the controls you use to move the characters around and make all the muscles squish up and stuff like that, and that was a lot of fun.  But I actually got into, like, internet stuff through environmental activism, especially climate change stuff. It's kind of hard to remember, but back in the day, it was not something people really had a big opinion about. It was very hard to get news stories written about this stuff if you weren't connected to a university campus or in a particular subculture. Like, climate change was just this super weird, vague concept. So we started using this fun thing called the internet to start doing mass mobilizations and media stuff, particularly with this guy named Bill McKibben, who turned it into this called 350.org, and eventually, that turned into the Sunrise Movement. Oh, I'm out, like, too old. [laughs]  So I moved on and then ended up getting into distributed operating systems stuff which was super nerdy. But basically, we didn't have very good platforms to build this scalable tech on. And there was Heroku, but that was about it. So I was working on an open-source version of a Heroku-like platform called Cloud Foundry. And then through that, got really, really annoyed at how difficult it was to observe what these things were doing and recreate it, especially because it was an operating system that we're shipping to all these big companies, and they're operating it, and then they're having problems. And I'm needing to recreate what it is that they're doing, and I can't just stand up their version of this system.  So I really became obsessive about stuff like tracing, which was not a big, popular concept at the time, and the ability for all of these different users who were running this thing who had their own collection systems, and they were using their own observability tools but being able to pump into all of those things. And that combo led to me becoming really interested in OpenTracing because it was like this perfect combo of, like, oh, it's distributed tracing, but it's just the interfaces. You can connect this up to any back end, and it really scratched an itch for me.  And the open-source projects I was working on around containers and stuff like that were kind of like a tire fire as far as the way the corporate backers of those companies were treating each other. It was also an opportunity to help build a new open-source community that sucked less. And so, for all those reasons, Ben Sigelman convinced me to hop over just this newfangled thing called Lightstep that existed for like a couple of months. So I joined the Lightstep train. I think I was lucky 13 when I got hired. ADRIANA: Nice. ANA: Did you join OpenTracing project? Because that's what was going on then, right? This is pre-OpenTelemetry. TED: Yeah. So before OpenTelemetry, there was a smaller project called OpenTracing that was much more narrow in scope. It was just focusing on tracing, and it was also just focused on instrumentation and interfaces. Like, the most annoying part of all of this, when you get into tracing instead of just logs and metrics, is that you have to propagate this context. And that is this huge lift for anyone who gets interested in this concept and just tries to do it on their own. They discover it's just this huge lift to do this context propagation, both serializing and deserializing this stuff and then shoveling it through the runtime of your program.  And so OpenTracing was designed to really just solve that problem for everyone without getting up in anyone's business about what they were going to do with that data. The scope of the project was a lot smaller. But while I was working on that, I became really obsessed with the idea that we needed to unify all of these different signals. It wasn't just about distributed tracing; it was like logs and metrics, and you could generate metrics out of your traces and all this stuff.  So I got really interested in this idea of having a unified observability platform because that was actually this other problem that I had, which was all of these data sources being disconnected from each other. And in order to do that, you really need to have a unified telemetry system. It works much better if all the data is coming out in an integrated fashion. And around that time, another project called OpenCensus had kind of come out of Google as well that was more focused on solving that problem. And the merging of these two things led to OpenTelemetry. ANA: I remember being at the KubeCon where they announced the merge and OpenTelemetry is a thing. And I was just like, this all sounds really cool. I hadn't really been too exposed to the space. I knew about the problems being in SRE, but I was like, okay, this all sounds really cool. TED: Yeah, I was really proud of everyone to be willing to merge all this stuff and create a standard. We needed to merge because having multiple projects in this space actually was working at cross purposes because this is this cross-cutting concern that has to go everywhere. So it's not like you can have different databases, and you can have your database, and I can have my database with instrumentation and all of this stuff. It really seemed to work a lot better if we joined forces.  And also, both of these projects were solving one half, like; OpenTracing really got it right around keeping this stuff stable and having the interfaces decoupled from the implementation. And it was kind of designed correctly in my mind for something that was going to be cross-cutting concerns, you know, sort of like how Log4j and some of these things work. Whereas OpenCensus was this like big ball of code, I was concerned about the way that was going to lead to maintainability and breaking changes.  [laughter]  Don't get me started on gRPC. And I felt like it might be going that same direction. But I really appreciated that they were doing the work of trying to build this client that sucked in all of these different data types. And it was also actually confusing to people using OpenTracing that we didn't supply a standard implementation. So it was really...it was like peanut butter and jelly. And I'm really appreciative of everyone who's working on both projects to be willing to come together. But I feel like we lost a year. [laughs] But at this point, it's all water under the bridge. AUSTIN: Yeah, what's a year between [inaudible 16:01] [laughter] ANA: What about you, Austin? How did you get started in technology and working in the observability space? AUSTIN: How far back do you want me to go? I was a very bullied child. [laughter] No. So I guess similar to Ted, I also...my shadow does not cast a straight line upon the earth. I've gone through a lot of different sort of career aspirations and interests and things like that. And then eventually, when I was in my late 20s, I guess I decided that at some point, you need to actually make money to survive and got into technology more professionally. I went into IT and did a lot of stuff with telecom and server break/fix kind of general IT stuff. But I always, you know, I grew up with computers. And I grew up learning how to, you know, write little programs in BASIC and being fascinated by how the computer worked and that you could make it do what you wanted it to do. And at some point, I went back, finished college, got into CS, and wound up at a startup called Apprenda. And Apprenda did platform as a service for .NET. And this was like early, you know, this is before Docker. This was before containerization really kind of took over.  I do very distinctly remember going to one of the early DockerCons and looking at the presentation and turning to my co-worker and being like, "Oh, we're so fucked," [laughter] because this was like what we did but better. So they, for a variety of reasons, went out of business. And in that transitory period, I tweeted out, "Hey, I'm looking for something in open source. I'm looking for something in this cloud space." Because one reason people might have heard of Apprenda is because we had the insight to purchase a small company called Kismatic, and Kismatic was the company that put on the first KubeCon. ADRIANA: No way. AUSTIN: Before they donated it to the CNCF.  ANA: Whoa. AUSTIN: And so there are actually very special editions of KubeCons like the one year where we ran, or we were part of the KubeCon organizing experience, I guess...there's a KubeCon t-shirt that has an Apprenda logo on the sleeve before they went away from putting sponsor logos on the shirts. But anyway, because we kind of were in that early cloud-native community, Ben Sigelman saw this tweet. And actually, that same day, I think I told Ted like, "Hey, Ted, go talk to this guy." [laughter]  And I remember sitting at home and getting on Zoom with Ted, and Ted pitched me on Lightstep and OpenTracing at the time, obviously. And my motivating factor there is really that, you know, I think you can't work in ops, in any kind of operational capacity with computers without just wanting to destroy all computers [laughter] because computers are inscrutable to our frail human senses.  And I had so many late nights and weekends spent just trying to figure out what the hell was going on; why is this working? You know, not only why is this not working, but why is this working? Here are my expectations, and then here's reality. And I got to know all this arcane stuff, and I have to manually piece together the execution of this distributed system across however many nodes, and it sucked. It absolutely sucked to do. So Lightstep was very appealing to me from that perspective.  I joined Lightstep. I was actually the first full-time remote hire. There was one other remote employee at the time, but she had started in person and then turned into remote. So I was the first fully remote. I think I was employee 40-something. ADRIANA: Nice. AUSTIN: I flew out to California, met everyone. It was great. And then I've been working with OpenTracing and then OpenTelemetry ever since. I think from the OpenTelemetry perspective; I come to it from a slightly different position than Ted. To me, it's less about the unification and more about the standards-making that really got me into this. Because one of the biggest challenges from an end user perspective is all of the toil and all of the duplicate of work that goes into just creating the ability to monitor a system, that's because there is no standard.  Every piece of software you use, unless you are in a very tightly vertically integrated stack like you're in the .NET world or something where there are a lot of conventions around this, and there is this provided monitoring libraries and everything, you're basically forced to go to some proprietary commercial solution, or some sort of community supported thing and ask them like, "Hey, can you please give me the ability to understand what I'm doing?" And I think that sucks. That sucks for the end users, and it also sucks for the implementers of those open-source projects, or those frameworks, or whatever, because they also don't have a clear answer. They can't serve their users well.  So what really has kind of attracted me to OpenTelemetry and one of the reasons that I work so much on the community side of OpenTelemetry is to push forth that standards-making approach to getting telemetry...making it ubiquitous, making it easy to use, so on and so forth. ADRIANA: And what have both of you seen since you started working in OpenTelemetry in terms of uptake, adoption, enthusiasm? I mean, obviously, we know all the major observability vendors are all in on it, but what about the people who are actually instrumenting their code? Are people psyched about it? AUSTIN: Psyched is maybe understating it slightly. I think people are ridiculously excited. I think it's actually almost to a degree that it harms the project in some ways because people are coming in so early and trying to use it in anger.  ADRIANA: [laughs] AUSTIN: And it's very similar to if you were around like in the early days of Kubernetes, and CoreOS, and Rocket and like that whole thing. We're at the beginning of the journey, not the end. And we're certainly not even, you know, we're maybe approaching a plateau in the next 12 months with OpenTelemetry where there is kind of a...the rate of change should somewhat decrease. But we're still in the super, super early days. And if you were around for Kubernetes and around for how Kubernetes developed, you know, comparing the Kubernetes of like 2012 to the Kubernetes of 2022 or whatever, maybe not 2012 but -- TED: Vaporware. [laughter] AUSTIN: But it was a thing where people were jumping into this with two feet, and they were jumping into an empty pool and getting their legs broken. [laughter] In a lot of ways, I feel like that's what's happening now with some people in OpenTelemetry is they're jumping in before the pools fill all the way up. And sometimes you'll be really successful with that, and sometimes you won't. It really depends on, like, there are too many qualifiers needed. Are you just using this one signal? Are you in this particular stack? Do you have a high level of operational maturity in order to build this stuff yourself?  Like, so it's exciting, but it's also terrifying because now we have a user base. Now we have people that are using this and relying on it. So that makes it much more difficult in some ways to iterate because you have to bring all those people on with you, too, right? ADRIANA: Yeah, it's interesting you mentioned that because I remember last year when I was working at Tucows, I was trying to get everyone on board with OpenTelemetry. And we actually had Ted and Liz come to talk to a bunch of the Tucows folks because they're like, "Well, you're saying we should use this, but we don't know if OpenTelemetry is ready." And so Ted and Liz came to basically do a Q&A to help, I guess, ease some of those concerns. But definitely, I can say that we're very excited to be using it and couldn't wait for it to move faster. And I can definitely see being on the other side of things how that can be extremely stressful. TED: Yeah, I was part of a really great end-user conversation today, which is a shout-out, by the way, to anyone who's listening to this. OpenTelemetry has a monthly end-user discussion group. You can find it on the OpenTelemetry calendar, which you can find, I think, on our website, and certainly in our community repo on GitHub.  But it is interesting to see how far ahead the users are of everyone and everything else. The biggest stuff we were discussing was people trying to do really complicated fleet management of their collectors because they're using their collectors to do just a crazy amount of work. I was kind of surprised at what people were using the collectors to do. People are building fairly advanced tail sampling pipelines and things, sort of just bolting stuff onto the front of whatever vendor observability tool they're using because those guys haven't built the features yet.  And there was this general complaint of, like, why is no one offering a unified observability platform? Why is there no place I can shove all this OTLP data into and get all of this cross-signal analysis and stuff? And I was like, well, you know OpenTelemetry is not even fully stable yet, and logging finally has just become stable. So we're basically just becoming stable. And we still really need to stabilize the data schemas. And prior to OTEL, there was no, as far as I'm aware of, no project, even like a proprietary project, that tried to collect all of this data and provide some kind of unified workflow across metrics, logs, and traces.  Like, how fast do you expect this product department to move in these companies? My prediction is two years. Two years from now will be enough time for at least some of these companies to produce a strong enough unified observability platform that the other companies go like, "Oh, crap," and kind of like wheel around to they're trying to do the same thing. But it's just interesting to see that people are really champing at the bit, whereas a couple of years ago, I feel like people were not even thinking about it, yeah. AUSTIN: I think it's interesting because the things that people are asking for, like, the tail sampling one's actually a really good example because I see a lot of people coming into the OpenTelemetry channels and the CNCF Slack, and they're like, "Hey, I'm trying to set this up. I'm trying to set this up. How do you do this?" And people aren't, you know, some of us are forced to respond. It's like, well, you've discovered why people pay money for this, right? [laughter] Like, you've cracked the code. Like, this is so hard, and like, yeah, it is. That's why people pay someone else to do it for them.  Because you're basically saying, "Hey, how do I set up this very complicated distributed consensus algorithm that is lockless and also can scale up to millions of QPS?" I think people have not a difficult time understanding it but a difficult time considering that observability problems as a space are really just as big and thorny and complicated as anything else. Like, it's a specialized sub-discipline that really, there's a countable number of people on the face of the planet that are like really, really super deep into this and really care about it. And most of those people are working on this for commercial interests.  So I don't know if we're going to see a move towards more things being pushed into the open-source world. I do kind of think that's the case. I feel like so much of the stuff that is right now proprietary and commercial is going to get pushed out. And speaking with just my OpenTelemetry hat on and not with my Lightstep hat, I think that's a good thing because the barrier to entry to create an observability product is actually very, very low right now.  I just saw, like, in the past year or so, there's been like two or three new ones that have come out that are just like, yeah, here's something where you throw your metrics in, your logs in, your traces in. And it runs on ClickHouse or some other sort of column store, and we slapped Grafana in front of it.  ANA: [laughs] AUSTIN: And you can do all this stuff. And it's true; you can; you can do all this stuff. And three years ago, that wasn't possible. So OpenTelemetry has really opened the door for a lot of the...perhaps the charitable way to describe this is innovation. Innovation in this space is possible now because we have moved that point of integration and shifted it left and shifted it more towards the providers of software or this open-source project to provide instrumentation and da-da-da-da.  The flip side of that, though, is we're in this awkward teenage phase, I think, on the analysis side. Everyone's trying out different things. Everyone's trying different clothes. And one week, you're wearing Hot Topic, and the next, you're wearing J.Crew. ANA: Oh my God, I love it. [laughter] AUSTIN: But with time, and as that shift-left continues to happen, then what you'll see, hopefully, is more commercial and open-source solutions that really narrow in on the analysis piece because that's the thing that I think we just don't do a good job of.  A buddy of mine works for a very small startup. And he's like one of the only support engineers they have. He was complaining earlier today in a Slack I'm in. And he's like, you know, so and so is having a problem, and one of the engineers says, "Oh, just go find the dashboard for it." And then he shows this video he recorded of him just scrolling through thousands of Grafana dashboards in a list. And there's like zero, you know, that's not helpful.  Go look at the dashboard is not a helpful solution to I'm having this problem. But that is what we treat as like, oh, this is what observability gives you. Now you can make those dashboards, and before, you couldn't make those dashboards. And I don't think that's the case. I think observability has to be more than you can make the dashboard now. It has to be about tell me your problem. And now, we have tools that can solve that problem for you or help you to solve that problem in a way that you couldn't before. ADRIANA: And I think that's why, like, it's equally important to promote OpenTelemetry but also those observability practices because otherwise, you fall into that trap of the wallow dashboards and relying only on logs for troubleshooting or what whatever other anti-patterns that so many organizations practice because that's all they know, right?  TED: But there really is room for the analysis tools to do a better job of improving our workflows. Like, I'm sitting in front of like five million dollars worth of computer hardware, but I'm trying to find correlations by looking at squiggly lines with my eyeballs. And I can't get around that really stupefying workflow until someone actually builds a computer program for me to just find the correlations and be like, here. ANA: I mean, we're starting to see that. I think those types of companies are starting to come up, and a portion of it is scary, a portion of it is cool. Like, it is exciting to see AIOps movement and just see a lot more of the automations on operations, reliability, and debugging. TED: I'm really curious to see people from other domains start to tackle this too. Like, there's this whole world of data science out there and big data analysis tools that are all about trying to find signals in the noise when you have 18 bajillion different dimensions on your data. That's like a whole practice that just has nothing to do with observability directly the way we practice it, but it basically does. And I'm curious; I'm waiting to see people take that big data scientists' toolkit and start piping OTLP data into it. ANA: Or even a way that the software is asking questions, and it's just pinging the answers to the collector or whatever you're sending OTLP to. Well, as we're getting ready to wrap up, we wanted to leave some fun things for our listeners to learn. We are all pretty distributed as a team. What are one or two things that you can't live without, whether it's a workflow in your work week or something you have at your desk at your home that just makes your day-to-day a little easier? TED: I'd say I can't live without sleep, but I've proven that wrong. [laughter] ADRIANA: Very fair point. AUSTIN: I think the thing I can't live without is my hats. It's very important to have a strong visual brand, and for me, that is hats.  ANA: [laughs] ADRIANA: So, what's today's hat? AUSTIN: Podcasting, famously a visual medium,  I'm holding up one of my signature black baseball caps, New Era size eight fitted. I actually buy these. I buy like four every year. ANA: You're not sponsored? [laughs]  AUSTIN: I know. Actually, yeah. So hey, New Era, if you're hearing this, hit me up.  ADRIANA: [laughs] AUSTIN: But actually, strike the name. They aren't paying me to put their name on, to advertise them. [laughter] But, yeah, I actually go through about four of these a year because I am really bad about remembering to bring raincoats or umbrellas or anything. So invariably, I will get caught in the rain with them, and then they get wet, and then they shrink, or the bill gets a little floppy, or I get sweaty in it, or whatever. It always looks like I have a new one because I probably do always have a new one. [laughter] Like, if you go two or three months without seeing me in person, then the next time, it'll be a fresh hat, so let's go with the hats.  ADRIANA: Nice. ANA: Do you have anything, Adriana? ADRIANA: My creature comfort for working; I guess I have a couple. I have a lava lamp at my desk. It's very relaxing to watch it do its thing. I have a little folding treadmill in my home office. So whenever I need to blow off steam and especially when it's cold out, and I don't feel like going outside like today because it's near freezing and there's snow, I hop on my treadmill, and I chill and watch some Star Trek: Voyager while I'm on my treadmill, and that relaxes me. [laughs] So that's my Star Trek shout-out, my super nerd out. [laughs] How about you, Ana? ANA: For me, it is my bright and colorful yellow Hydro Flask that keeps me hydrated throughout the day. But at the bottom, it has this cute dodge on a freaking piece of wood typing on a computer sticker that got given to me by Denise Yu. So every time I'm on calls and drinking from it when I see my face, like, it just makes me happy, which is always like a nice, little thing to have in your work there.  ADRIANA: Aww. [laughs] ANA: And I hope I brighten people's days, too, with a cute sticker. [laughs] And then my blanket, just, like, I'm always cold. So I just kind of always seek a little comfort while I'm at my desk.  ADRIANA: That's awesome.  ANA: And I have two more questions that maybe we'll get through. So we all recently got to meet at KubeCon. But for some reason, even though we're a DevRel team, no one thought about taking their phone out and snapping a picture. So how did it feel to finally meet your team in person, as we never all got to be in the same place until now? AUSTIN: Everyone except Ted is shorter than I thought they were.  [laughter] ANA: Thank you.  AUSTIN: Sorry. ANA: We have actually said it on this podcast. Adriana and I are really proud to be short Latinos. [laughter] AUSTIN: In fairness, I had met everyone in person except Adriana before. So it's also one of those things where I'm the tallest person on the team. It's a little like, oh, I'm the tall person again. ANA: [laughs] ADRIANA: You're quite tall. I didn't expect you to be so tall. [laughs] AUSTIN: I think a lot of us, I mean, literally all of it is the perception you get of people when they're on webcam, right?  ADRIANA: Yeah, it's true.  AUSTIN: And I have my camera set up explicitly at eye level. So if I'm doing, again, podcasting, famously a visual medium, but like, if I sit up in my chair and I'm looking, like, I am eye-level with the lens. I don't know what perception that gives people about my height. There's no angle, right?  ADRIANA: Yeah, yeah. AUSTIN: But a lot of other people, it's either camera pointed up, or camera pointed down or something. And so you just get like this thing it's like your mental model of someone's height is completely based on that. ANA: You used mental model in a non-architecture way. And my mental model is breaking.  [laughter] AUSTIN: That's my value add. Obviously, the reason we didn't get a chance to take a picture is because who would have held the camera? ANA: That's true.  ADRIANA: I know. We could have done a selfie. AUSTIN: We could have done a selfie. I have long arms, too, so. TED: One of the 8,000 people walking around [crosstalk 37:39] AUSTIN: Also true. ADRIANA: It never even occurred to me to take a picture, or it was always the thought of, like, we'll get to it tomorrow. AUSTIN: We were all pretty busy. ANA: Yeah, that is true. ADRIANA: Yes, yes. Very much. ANA: Well, for the next one, we'll snap some photos and post them for our podcast listeners to get a chance to see some of the brains behind the podcast. AUSTIN: Next year in Amsterdam.  ADRIANA: That's right.  ANA: Yes.  ADRIANA: Yes, woo-hoo. ANA: Last question before we wrap up the episode. If you were to quit technology, what would you do? TED: I would start making more short films. I'd get back into cinematography.  ANA: Nice. ADRIANA: That's awesome.  ANA: Austin. AUSTIN: Hmm, I don't know. Is there a job where you get to destroy computers? I guess I'd be e-waste recycling; there you go. Destroying computers all day. [laughter] It would be very cathartic. ANA: What about you, Adriana? ADRIANA: Oh, man, I'm trying to think because I kind of did that a few years ago where I quit tech temporarily to become a professional photographer and realized that that wasn't for me, so now I don't know. I think I'd just...I don't know if I could leave tech because I feel like I'm unproductive if I'm neither digging into tech nor writing about it. So I don't even know what I would do with myself.  ANA: [laughs] ADRIANA: Maybe I'd be rock climbing a lot, [laughs] maybe that's it. Find me on the rock. Well, maybe I'll actually go outdoor rock climbing at some point. [laughs] AUSTIN: I actually have a serious answer for this question, by the way, that I just recalled. My serious answer is I'd probably go back into academia.  ADRIANA: Oooh. That's cool. AUSTIN: I enjoy doing research. I enjoy writing. I enjoy teaching, so. ANA: Nice. ADRIANA: That's awesome. ANA: I feel like, for me, it would be like a travel photographer, but someone that does only ocean stuff, like, just being on the water will call my attention the most, yeah, because that would be leaving tech. If I got a chance to stay in tech, I would do my own startup, but that's a whole other thing.  TED: Nice.  ANA: Sweet. Well, that gets us to wrapping up our episode. Thank you, Ted and Austin, for joining us today in the episode of getting a chance to meet the Lightstep DevRel team. Thank you to all of our listeners for joining today's episode. We loved talking a little bit more about OpenTelemetry, getting a chance to talk about caffeine, and just about where we see observability going. Don't forget to subscribe and give us a shout-out on Twitter and other social media platforms @oncallmemaybe. Be sure to check out the show notes on oncallmemaybe.com for all the links and resources we've talked about on today's podcast. And don't forget to connect with us and our other guests on social media. For On-Call Me Maybe, we're your hosts, Ana Margarita Medina. ADRIANA: And Adriana Villela with... Ted and Austin: Peace, love, and code. ADRIANA: [laughs] Love it. AUSTIN: Great job, everyone. ADRIANA: Woo-hoo. TED: Aloha.

About the guest: Riaan is a Principal Consultant with Servian, based out of Brisbane, Australia. Originally from South Africa, Riaan has worked for multinational companies in China, Portugal, Germany, the US, and Australia. He has a keen interest in Automation, Infrastructure as Code, and Configuration as Code, with a strong focus on DevOps ways of working. Riaan is also a HashiCorp Ambassador. In his spare time, Riaan enjoys hiking in nature, camping, trail running, and motorcycles.  Find our guest on: Riaan's Twitter Riaan's LinkedIn Riaan's GitHub Riaan’s Website Find us on: On Call Me Maybe Podcast Twitter On Call Me Maybe Podcast LinkedIn Page Adriana’s Twitter Adriana’s Mastodon Adriana’s LinkedIn Adriana’s Instagram Ana’s Twitter Ana's Mastodon Ana’s LinkedIn Ana's Instagram Show Links: Servian HashiQube kubectl MiniKube HashiCorp Ambassador  Terraform Terragrunt tfenv DBT Helm Docker  AMD64 Processor AArch64 Apple M1 Processor Apple M2 Processor Vault SRE / Google SRE Book (aka “SRE Bible”) Spring Boot  Google Borg Lightstep New Relic Werner Vogels (VP & CTO at Amazon) Additional Links: Blog: Running HashiQube on Multi-Arch (Arm and x86)/ Multi-OS (Linux, Mac, Windows) with Docker Desktop and Vagrant Transcript: ANA: Hey, y'all. Welcome to On-Call Me Maybe, the podcast about DevOps, SRE, observability principles, on-call, and just about everything in between. Today we're talking to Riaan Nolan, who's going to be sharing some really cool tips for us today. And we're very happy to have you join us today. RIAAN: Thank you very much. Good morning, Ana and Adriana. Nice to see you, guys. And thank you for having me on your podcast. ADRIANA: Oh, we're super excited to be talking to you today. ANA: So the first question we like starting a podcast with...since it's early morning for you, and it's midafternoon for me, and almost evening for Adriana, what's your drink of choice for today's podcast? RIAAN: Well, it's 7:00 o'clock here by me, just after 7:00, so I'm having coffee. But normally, I love beer, anything cold, since I live in Brisbane, Australia. It's normally quite hot on this side. But it's coffee today. Cheers. ANA: Cheers. I got carrot and orange juice. So it's somewhat tasty. ADRIANA: Nice. Got plain, old water. [chuckles] ANA: Gotta stay hydrated with whatever heat's going on in this world. Super sad panda for this global warming. RIAAN: Oh, it's summer by you guys now. ADRIANA: Yeah, yeah. We had a heat wave here last week in Toronto where we got hit with like 40 degrees Celsius, so 100-ish Fahrenheit, so that was very unpleasant. And my AC broke in the midst of it, so boo. We got it fixed, fortunately, but boo. RIAAN: Wow, that's cool. I can't wait to have that weather. I'm such a fan of hot weather. Really, I love it. If it's 38-40 degrees Celsius, I really love it. ADRIANA: I do dig it as well, having grown up in Rio de Janeiro, but it definitely got very stifling inside the house. [laughs] But yeah, I'm for warm weather as well. I do enjoy a nice balmy temperature. ANA: It's really weird for me because the entire United States is having temperatures of 80 degrees, 90 degrees-plus, and I reside in San Francisco, California. So sorry I use Fahrenheit, even though I grew up with Celsius; [laughter] that's a disclaimer. And secondly, San Francisco has been really cool, like not freezing cold, but it's been chilly, like 50-60 degrees Fahrenheit. And I go on the weather applications, and the entire United States is red.  And then you look at Spain and France, and they're having deaths due to the heat. And I'm just like, I am so scared for whatever is going to happen overall in the world. But when you're one of the only cities that is not going through the heat wave, I'm like, so what are we going to get? Like, we're going to get something. What is it going to be? ADRIANA: It's coming for you, I think. RIAAN: I think it's that La Niña, the opposite of the El Niño at the moment. And that's what makes the weather a little bit lopsided, a little bit weird. ANA: And it's interesting because it's like, weather is one thing [chuckles] that I refuse to try to get into and try to understand. I have the understanding of how water precipitation and condensation hurricanes work. But then it's like, my complex systems start at DevOps and SRE. [laughs] I refuse to start an entire other study of how other systems work. [laughter] ADRIANA: You've got enough complexity in your life.  ANA: I think so.  RIAAN: That's funny, you know. ANA: So as we were breaking into having these conversations, I actually want to bring up the amazing conversations that happen virtually in COVID where I think we're geeking out a lot more. We have a lot more virtual calls. And it could be similar to the hallway tracks. We have conferences and meetups, but it's really awesome to be able to share tools.  And we got a chance to share some of our favorite tooling and talk about Riaan's open-source projects. So I'm going to give you the mic for you to share a little bit more of what you have actually been working on and contributing to open source. RIAAN: Thank you. So as you know, I'm a HashiCorp ambassador also with Adriana, my colleague. ADRIANA: Woo.  RIAAN: I'm HashiCorp staff. So the one tool I've been working on is HashiQube, and it runs all of the HashiCorp tools and MiniKube, and it now runs in a Docker container. So I'm really excited about that. It helps me a lot when I have to proof of concept stuff up. Sometimes just put up a...what do you call it? The Ansible Tower running on top of Minikube, and then I can use that to connect to Vault to sign certificates or get secrets. And if I have to use Docker or if I have to Terraform something up, it's just a little tool that I can quickly use to do that. And then, on the other side, there's a little tools container that I've been working on that's on my GitHub. And that's just a little container with Terraform, and tfenv, and Terragrunt, and DBT, and kubectl, and Helm that you use for data projects or infrastructure projects. And it's just a little container that I try to get it to run on AMD64, and AArch64, the M1 chip, so the M2 Apple chips. So that's how I keep myself busy with and tinkering with that. ADRIANA: That's awesome. I'm personally a big fan of HashiQube. And I think that's how we connected initially.  RIAAN: Yes. ADRIANA: And it's funny because I found out about HashiQube from a former co-worker of mine when we were doing some local Hashi setup. And he told me about HashiQube, and then I started playing around with it. I'm like, oh my God, this thing is the coolest thing in the world. Because I can now mess around with all the Hashi tools without having to rely on some Nomad cluster running in the data center.  And for me, that was amazing because that meant I could be in control. (I like being in control.) I didn't want to have to rely on someone. And we were running...we had our own private cloud. So it wasn't like I couldn't just as easily spin up a virtual machine the way that I would if we were on a public cloud. So for me, this was an amazing salvation because I was able to tinker. And I also want to say, as a plug for HashiQube, the stuff that I did in my local environment in HashiQube pretty much translated when I moved stuff over to deploy in the data center, so super stoked for that. So yeah, I can't say enough good things about HashiQube. And I've blogged a lot about HashiQube as well in my Nomad explorations. RIAAN: Thank you. ADRIANA: So awesome. Check it out, check it out. [laughs] RIAAN: Thank you. I'm glad it helped you. It certainly helped me when I was sitting on the train or trying to learn a new thing here, you know, seeing our Vault can connect to MySQL and create database users and seeing that lease expire, that time to live. It's fascinating and so easy to do now. You just spin up a MySQL container with some arguments, and there you go in and plug it into Vault database engine, and off you go. So, yeah, it helped me also a lot. ANA: We have so many tools out there. That's always nice when we have easier ways to onboard, especially when it comes to making the space that we work on better. That actually made me think about...I know both of you are huge open-source contributors. And I know some of our listeners are too. What was your biggest reason for going into contributing to open source and especially starting new projects for the ecosystem? RIAAN: I've always been interested in open source, and I've always been using Ubuntu. But something that I enjoyed doing is also contributing to open source and helping other people learn. And I saw one of your questions is how do you cope with failures? And this is so much related to this topic because if you contribute and you can push to GitHub, and other people can pull your changes and test it out for you, and work together on this, then it kind of helps learning and dealing with failures a little bit easier.  Because every single day, you have that aha, it's working moment, that little bit of happy endorphins in your brain that just fires, and you get it ready, and then you can make that commit. And sharing your work with other people, it's really enjoyable. It makes life interesting and fun. ADRIANA: I was going to say when you put your work out there, when you open source your work, you are basically making yourself vulnerable to the outside world. The reward is immense because you can have people who come in and start using your product. And they're like, oh, this is awesome, but here's a way to make it better and contribute, and you help each other improve.  But it can be so scary too because you open-source something, and then you can have some jackass go, "Well, this is crap. I would do it much better if it was me, and this was what I would do." And so it can be really scary. So I think it's so important to put yourself out there, even knowing that that could be the reaction that some may have from putting your work out there in public. RIAAN: Totally. I mean, you've seen the way I do HashiQube. It's a bunch of Bash scripts and Vault commands and stuff. It's nothing really fancy or anything, but it works, and that's version one. And if people can make it better, then even great. For me, the intention of putting it out there is to help other people, and share the stuff, and get things going easily, being able to test new versions quickly, or just get something up.  And I always look at it from that perspective, if your intentions are pure, then haters are going to hate, and there's nothing you can do about it. Haters will hate. And I try to just live on the bright side of life. And if they can do it better, then great, help contribute. If you can't do it better and you want to slam me or whatever, that's also okay. Some people sometimes just have a difficult day, and they bark at you.  And coming out of this COVID thing, you can definitely feel the stress levels. It's been a little bit on the upper side of life. Of course, mental health awareness, we just had Mental Health Awareness month, I think, last month. So you can definitely feel people are sometimes on the edge. There is war in the world. We haven't had war in such a long time. And now it's there full-blown in the open. And, of course, the media want to capitalize because bad news sells. And this is not good for anyone.  So what can you do? You got to just look on the brighter side of life and stay true to yourself and do what makes you happy and try to help other people. What else can you do? And plus, if people say you could do it better, I always tell them, "Man, I come from Africa, and if you think that I'm the best out there, you're the idiot. Africa... you are the first world; if you want to make it better, make it better." [laughs] Just a little bit of a joke. Us Africans are actually super clever, but just a joke. [laughter] ANA: I think it's also, as you say, the world being kind of really crappy right now with folks just trying to push through as much as they can. I think I've heard the sentiment from a lot of people that open source was just their outlet where it was their only way to cope with all this idle time that they had or the lack of in real-life connections like family and friends or health reasons you can't really see anyone.  Being able to connect with one another and build a better ecosystem really, really pushed people. There are a lot of pros and cons. You need to have a lot of psychological safety within an open-source project to be able to have a code of conduct that it's if you're an asshole in PRs, [laughs] GTFO. You can't be there, and the organization is going to hold you accountable. They'll go to bat for you, and code of conduct breaches will get put in into a repo. RIAAN: It will, you know. But I could think of nothing more exciting than learners telling you, "Your PR sucks." [laughter] I would frame that, blow it up, and put it on my wall. Look, learners told me my PR sucks. [laughter] It comes with the pressure of stuff. Sometimes these projects do with all of the good intentions.  With all the time we've been trying to make stuff simpler, I just feel that it's been getting more and more complex, and complex and more complex. And as soon as you look through the microscope lens and you look deeper, then there's more to see. And I think we will get there. We are definitely still on the journey. I saw a tweet from Kelsey Hightower the other day saying DevOps is dead; long live DevOps.  ADRIANA: [laughs] RIAAN: And isn't that so true? Most companies we work for are only now doing this transition into DevOps and the ways of working, and we've been in this so long that it's already kind of last decade for us. But still, it's still so fresh in the minds of corporates that it's actually fascinating. ADRIANA: Yeah, it totally is. It's funny because I think Ana and I were talking about that earlier with regards to DevOps at the enterprise level is so different from DevOps at the non-enterprise level. Because at the enterprise level, you are dealing with bureaucracy, red tape, moving parts.  You've got the lifers who do not want to change. You've got the VPs who are basically marking their territory and also are going to be resistant to change because you're encroaching on their little kingdoms. It is such a complex sociotechnical problem. I don't even know that there is the solution for it.  I think the only best approach to start taking with that is you need some sort of like, honestly, I think you need the CEO on board with this stuff or the CTO. They can push that down to the lower levels, basically providing that psychological safety, if you will, to the other folks saying "Yes, it's okay to follow DevOps practices." But it's so hard to get up to those levels because those guys are so far removed that how do you have that conversation? I mean, even the ones that were technical have been so far removed from tech. RIAAN: What I think we're going to do is we're going to outlive them. And the way we are doing it, we are already prepping the next generation, the next wave of DevOps people. So as we get older, we're also going to move into management, but this mindset, this curious, this forgiving, this no-blame culture, I believe, is going to stay with us.  Let's say you go up into management, CIO level, CTO level, you will still have this foundational layer of DevOps in the fiber of your being. And you will be naturally more accommodating to change that bubbles up from the bottom, you know what I mean? So I just think it's a matter of time.  We should just keep on spreading the good word of DevOps, and eventually, we're going to outlive them all and make this the true gospel. Eventually, this is going to be the gospel because this is the way. I don't want to sound [laughter] [inaudible 16:15] or something. But this is the way, you know, this is the way. It starts off small.  And people say DevOps is just for tech. But let's say marketing have a huge campaign. Now, SRE sees the lights go off, and the alarm goes off. These clusters of fleets are under pressure. They're getting extreme traffic. Where is this coming from? Is it a DDoS attack? No, it's legitimate traffic. But no one told SRE that they're planning to have a marketing campaign. Or if someone deployed something, then put a New Relic marker, a deployment marker in this APM graph that can say, well, I've just deployed this version. And now we can see the performance either dip or change or the error rate. It could be just as simple as putting a little bit of a marker there or a shared calendar. And let's say we deploy some new Terraform code or something like that; put a marker.  I love New Relic graphs that are full of little lines that you can see what is happening there. And people always hate chatty Slack channels and stuff. I actually love it because it makes it so easy to go back and see, oh, wow, here the wheels started coming off. And, look, this is where it caught fire, guys. Check here. It's burning, you know? [laughter] And it could just be so simple. ANA: It's true. And it's like, this communication, like, you're right, it's not just Dev and Ops. It goes above that marketing, communications, and just being aligned with your sales org. Breaking that silo within a company is going to make you a stronger company. And it's also that over-communication is not a bad thing. Like, yeah, there are some relationships that might not be the best with over-communication.  But as we think about human society and the way that things are building and going about it, over-communication is allowing us to express ourselves and really think about our thought patterns of going there. And it gives the ability for people to ask questions. And when we think about the world that we're building, whether it's in tech or as a society, we're building off the foundation of someone else and just trying to make things better for the next person. RIAAN: Isn't that the truth? You said it. That is the whole idea behind infrastructures as code, versioning, tagging, and releases. And QA and testing, we're just trying to make stuff better. ADRIANA: Yeah, that's the thing. And I think those of us who see the magic we're like, duh, of course, it's going to be this way.  ANA: [laughs] ADRIANA: And then we feel compelled to spread the good word of DevOps across the board because we're like, can you not see that there is no other way?  [laughter] RIAAN: And it could just be some people want to use containers, and some people want to use VMs. And the tech around it doesn't really matter.  ADRIANA: No. RIAAN: The tech doesn't matter, really. If you have virtualized something and you haven't yet gone on your journey of containerization, I'm not going to burn you for that. I mean, you're still on your journey.  Maybe you have a flipping monolith with persistent data that you need to carry around, like “The Hunchback of Notre Dame” or something. You can't help it that you've got a hunchback. You've got to carry this thing around. But you're just on your journey, I think, I suppose. ANA: Yeah. And I think that's one of the things that I've been thinking a lot about. I was talking to Adriana last week, where it's like, I love a lot of the thought pattern that has come with SRE because it's an implementation of DevOps of making things better for the purpose of reliability. But the more I started thinking about the space, it goes back to communication and the people aspect of it, where you are leading with empathy. You're understanding that a lot of people have come together to build a complex system. And guess what? [laughs] It's very large and very complex. Once you really look under the hood, you're like, oh, snap.  And if we're able to understand that there's going to be a lot of contributing factors into things going wrong, whether they're incidents or not, then we really can start over-communicating. Or I recently joined our engineering ops channel at the company that we work for. And it's been really nice to start seeing everyone's deployment status. It is word for word I'm deploying this, and then they'll copy all their kubectl, like, export before they roll out the deployment.  And it's like, by having that visibility, let's say something does go wrong, we have a back trail. And this just adds on to a lot of the other work we do where it's like observability, getting a chance to view those traces of how your system comes together. It's just like, this is beautiful. [laughs] Like, let me present you my large complex system that we can know what went wrong.  ADRIANA: Yeah, it's so true. I love that idea because I do feel like, you see, in a lot of large enterprises, there's often the dude who's the keeper of the scripts, and they reside on his computer, and that's it. And DevOps encourages us sharing is caring. Let's make sure that everyone knows where these frigging scripts reside. Because if shit hits the fan, now you know, okay, it was this script. Okay, let's go look into this script and figure out what the hell happened. It's that sort of open mentality that really helps us become better, right? RIAAN: When shit hits the fan, it's nice to know how fast that shit was flying, [laughs] how that shit hit that fan. It's good to just have a couple of seconds before and after. And it's lovely, actually. It's a very unique time we are in, and this movement towards observability. I liken it to you've got a TV and multiple channels. And you've got your radio, and you've got newspapers. But if you have all of those on at the same time, sure, it would be a little bit busy in your house because of the kids, TV, and the games, and whatever.  But if you could just have one channel on when you want it on, or you could dial into one channel to see this stuff...so people who say, "Oh, it's too noisy," or whatever, well, I can't exactly tell the newspapers to stop printing because it's too noisy for me. They're going to think you're a freak. Turn the channel off if it's too noisy, or look the other way if it's too noisy for you. But the thing is, we need to get it noisy so that we have this observability, this insight.  And in the future, I think we're going to get maybe a little tablet injected into our arm that can measure our cholesterol. And on our phone, we can see a little graph that shows you oh, wow, look, in three or four years, I could have a heart attack. Or holy moly, my blood pressure is really high, or my oxygen levels just tanked, or something like that. I mean, I think it's just preparing us for the next thing; it's observability. ANA: You literally nailed one of my niche spaces of bioinformatics and mental health because that is literally my biggest passion project that I hope to one day start developing. It's like being able to aggregate a lot of these metrics. And this just comes from my SRE mindset where it's like really complex systems on architecture, tech space, but then really complex architecture in our bodies because of the amount of microservices that we have to run to literally function and wake up. [laughs] RIAAN: Yes. There you said it. Exactly, there are a lot of microservices. ANA: It's just like, it's the most beautiful, like, changing your mental model to go into a passion, and it's literally the biggest passion project that I have of trying to make a dent in this. Because it's that, like, if we were to have a lot more understanding into the ways that our brains are processing thoughts, processing...trauma is a lot harder to get like bioinformatics on until we do more research. But just going from habits to things that align to your values to just like, why is my sleep pattern so bad? Maybe I should look at how late I'm eating and tracing [laughs] what your actions were towards your entire day.  RIAAN: Very, very true, you know. ADRIANA: To your point, Riaan, about people get really frazzled, I guess by information overload, but sometimes you don't know what information you need. So you'd almost prefer to get as much information as possible to identify the information that's useful to you so that you can at least make informed guesses, informed decisions around what's going on. Because if you don't have enough information, well, that's worse, I think, because -- RIAAN: Yes, you said it. You said it. You don't have enough, so you just see when the box died. You don't know what happened to the CPU, or the RAM, or the disk space tanked, or whatever the heat. You just see it went down. You have no real information to work with. ANA: It's like there are so many things that need to happen to keep something alive [laughs] that it's that; it's like, was it a people process that didn't get followed? Was it the technology timing out on a certain action that it was going to take to remediate this problem? Or was it just a lack of awareness that this can happen? We don't have enough understanding of the stack that we're running, or we didn't really do the proper types of capacity testing to sustain the marketing launch that we rolled out last week. RIAAN: Exactly. But this data is important, especially in this new world where we are going towards. And outages affect companies. They affect bottom lines, and sales, and people's perception about the company. But the thing is, it's unavoidable. It is just unavoidable.  So Dr. Werner Vogels from AWS said (What a charismatic guy.) things will break. They are going to break. And if you don't plan for failure and you don't work around this for this expected failure, this is the way of the new world, The New World Order. [laughs] ADRIANA: Right? Yeah, [inaudible 26:39] RIAAN: [laughs]  It's finally here. Outages. ANA: It's perfect because I'm literally giving a talk on incidents on Friday. And as I was going through my slides yesterday, I threw Werner's quote of everything fails all the time. We have to take that understanding that the technology that we're building is so fragile that if you don't come in with a hypothesis that things will break, [laughs] you're going to be really mad when they break. ADRIANA: Yeah, yeah. Yeah, and that's the thing. I think some people have this expectation that things are going to work perfectly all the time. And as consumers, we're like, ah, this frigging website's so slow. What the hell's going on? And it's easy to forget that even that little blip, the fact that your system recovered from that tiny little blip, is because there was a group of SREs working behind the scenes really hard to make sure that your system is reliable.  And I think it's good to remind ourselves of that, that we should be grateful for the little blip because we know that it could have been a big blip instead. I mean, our systems are so complex these days. It's ridiculous. I think our smartphones are more powerful than the computers on the Apollo rockets back in the day. It's mind-blowing. So yeah, like, you're going to get a blip every so often because nothing is perfect. RIAAN: I was reading just going through this blog post on Kubernetes for clients with putting a Spring Boot onto Kubernetes. And I read that, and then they mentioned that Kubernetes started from Google Borg. And then I clicked on this, and I got that PDF. But at the bottom, I think it was on page 18, they said the user experience...and I'll copy it for you here about SRE, so I'm posting it for you now. And I actually thought that it's actually so nice.  It says the user perspective SREs do much more than system administration. They are the engineers responsible for Google's production services. They design and implement software, including automation systems and manage applications and service infrastructure and platforms to ensure high performance and reliability. And if you just think an SRE, that's like three letters.  ANA: [laughs] RIAAN: But what they actually do on a day-to-day basis [laughs] is incredible. It is just incredible the amount of effort and thought that goes into it. ANA: And it's really interesting because the more you start talking to folks about their SRE practices at enterprises and startups, you still realize that every single company implements SRE so differently. I'm a huge fan of telling folks to read the SRE Bible, the Google book where they started talking about this practice that they've been doing for ten years for production excellence.  And you start realizing that every single company is less complex, more complex, has different leadership, has different silos that they need to work on, or their stack or hours that they have are so different. They're like, you can't really adopt word for word what the SRE model from Google is, and I really love that. I feel like a lot of SRE is leading with empathy, and it's that of constantly restructuring your practices to meet the demands of your employees and the ways to make them more successful.  But it's also been digging into the principles of DevOps of iterating and learning that we're constantly able to be like, I work at Lightstep; this is how we do SRE. I work at New Relic; this is how we do SRE. And everyone's implementation and problems are so different, which is so freaking cool. ADRIANA: I think, too, to reinforce the point, I think reading the Google SRE Bible is important. But companies who want to stand up SRE teams need to also ensure that they cannot possibly run their SRE team like Google does. It's like, okay, take the best ideas from that book that will suit you. Maybe make up your own shit too [laughs]; that's going to work for you, and that's going to be what SRE means to you. But you get the general idea, but I think each company has to craft their own journey. RIAAN: Totally, and there's so much we don't know. Sometimes I feel like marketing...at some stage of the game, everyone thinks they are the most important, every department. And it's kind of that egocentric outlook on the world and business and what we do that hampers collaboration in a way. If you want to really collaborate, you need to understand that other people have built the roads that you drive on. And for it to work, you need to build something else for someone to drive on. It is really that generational working together and building on previous gains that makes stuff better for everyone. ANA: It's like how to be human type of learning. [laughs]  RIAAN: Yeah, in a 50-minute podcast, learn how to be human. Super funny. [laughter] ANA: It's just like running a production service. That's easy.  RIAAN: Isn't it? [laughter] It's so easy.  ANA: As we're coming to the close of the podcast, what practical advice do you have for our listeners today? RIAAN: Don't worry about failure. Fail fast if you do fail. None of us are superstars or anything, so your name doesn't mean shit anyway. [laughter] So put your code out there. If people think it sucks, it sucks. If they like it, they like it. The best thing is just paint your picture. Do your thing and put it out there because that will help you grow.  My colleague, when I first started at Servian or took my journey onto Terraform he said to me, "If you want to get good at Terraform, write as much Terraform as you can." And that is where, you know, if you want to get good at something or you want to get better at it, write as much as you can. Just write Terraform. Just go out and write stuff. If you want to get good at something, go automate it and try it again and again.  And maybe today it's a Bash script running in a...it's a Bash script, and then it's the same Bash script but running in your CI/CD pipeline, and tomorrow with a better pipeline, and then it's a kubectl running stuff on clusters. No more Dockers or VMs or whatever. But if you don't start that journey, you're never going to get to the end of it.  And the sooner you start, the quicker you're going to have fun and enjoy yourself. And just put yourself out there. It doesn't have to be perfect. Day one, even God destroyed the world. [laughter] He was like, this version one ain't working for me. [laughter] Get the water. Bring me water. [laughter] And so it's okay, man. It's totally fine. And I suppose that's the advice I can leave you with and also, don't litter. [laughter] ANA: Those are good ones.  ADRIANA: I love it. I love it. RIAAN: So thank you very much. It was so lovely to chat with both of you, really. What a wonderful way to start my day. And, Adriana, thank you for even thinking of me for this podcast. I've really enjoyed it. ADRIANA: Oh, thank you. We had such a great time chatting with you. And I am such an admirer of the work that you do on HashiQube. I do encourage our listeners to check out HashiQube because it's got some really awesome stuff. And it runs also on both Intel and M1 architectures, so super bonus points.  And the two articles that Riaan was talking about earlier, we'll post that in the show notes as well, so for anyone who's interested in checking those out. And we will also share your social links in the show notes in case anyone wants to reach out to you to ask about HashiQube and other things. ANA: And don't forget to subscribe and give us a shout-out on Twitter via @oncallmemaybe. We're going to be posting a lot of cool, little snippets of some of the guests that we have this season. And make sure to connect with us and our guest. ADRIANA: And signing off from On-Call Me Maybe I am Adriana Villela with my awesome co-host... ANA: Ana Margarita Medina signing off. RIAAN: And peace, love, and code.