Why Do Engineers Hate Security?

There is a relationship between security professionals and engineers. Explore the possibility of engineers disliking security personnel and how security professionals can improve their relationship with engineers.

Security professionals need to be empathetic, have strong soft skills, and be able to influence and embed themselves within the engineering team. Resource management is essential, and avoiding engineers feeling like security is always giving them an over-the-shoulder look. 

Being part of the engineering team and understanding their world is vital to being a security professional that engineers don't hate. It is challenging to sell security as insurance to engineering leaders who may not see the value in investing time and resources.

FOLLOW OUR SOCIAL MEDIA:

➜ X: @SecTablePodcast
➜ LinkedIn: The Security Table Podcast
➜ YouTube: The Security Table YouTube Channel

Thanks for Listening!

Transcript

Copy Transcript Copied

Chris Romeo 0:09

Hey folks. Welcome to another episode of the Security Table. This is Chris Romeo, joined by Matt Coles and Izar Tarandach. And the topic for today is the sound of silence. So We will literally just sit here silently for 15 to, no, we couldn't. There's no way we could be silent for 15 seconds or 30 seconds either. But, uh, we were joking about the sound of silence and I was made, it was made aware to me that there is a remake of the Sound of Silence by Disturbed. So this is not a music podcast, but we figured we'd start there and I have a feeling we're gonna come back to the sound of silence. I think we can tie it in at the end to what we're gonna talk about today. So, question I'm posing for Matt and Izar and I guess the rest of the universe as well, is. How can you be a security person that engineers don't hate? How can you be a security person that engineers don't hate? So let's, let's unpack the, the, the second half of that. First, like, do engineers hate security people?

Izar Tarandach 1:13

I think that hate is a strong word. But it's useful.

Chris Romeo 1:22

I thought you were gonna say no. It should be more like dislike. It should be a soft No, you're like, it's a hard, it's a tough word, but it's

Izar Tarandach 1:29

it's, yeah.

Chris Romeo 1:30

point.

Matt Coles 1:30

We, we need, we need a chart cuz there is, there is most definitely a non-zero percentage of hate going on It's, it's, it may be small of the grand scheme of things, but it's a word. It's an important word. is not inconceivable

Izar Tarandach 1:47

it's,

Matt Coles 1:47

hate

Izar Tarandach 1:48

it's another, it's another one of those things that you go, it depends, and it depends on what kind of security person, like are we talking compliance, because then even security people hate thinking compliance people. But

Chris Romeo 2:09

True, true.

Izar Tarandach 2:11

not personally,

Matt Coles 2:12

and there's, I mean, there, there's different levels of hate, right? There's, there's, there's, oh, he's coming. Oh, I gotta do something. Now, this person is evil. Why is he, why is he or she here? Uh, Get outta my, get outta my office before, before I throw something at you

Chris Romeo 2:29

Well there, no, there's one before that. That is turn out the lights and maybe they won't think we're actually in here.

Izar Tarandach 2:35

And There's one.

Chris Romeo 2:36

Everyont be quiet. Lights out.

Izar Tarandach 2:36

There's one after that. That's the voodoo doll right?

Chris Romeo 2:42

So why though? Why, why do engineers, let's say it's not, I mean, we shouldn't use the word hate, like, let's say dislike, let's just soften it a little bit so we get more, more tens of percentage points that

Izar Tarandach 2:53

So,

Chris Romeo 2:54

the, in the graph here. So, but why, what, what, what does security do in, in both of your experience, that makes engineers want to run away and hide and, and, and find ways to not deal with us?

Izar Tarandach 3:06

So per personal, personal story time, uh, once had the team. That circulated an email saying, stop talking to Izar. Because every time that we give him more details, he finds more problems. Right. So it's not like there was hate in there. I mean, nobody went with, perhaps there was, and I, I wasn't aware, but the, the, the fact of the matter is that, uh, I, I think that there is one thing that we can agree, right? That, uh, Maror and dium apart, the one substance in the universe that's the rarest and, and most costly is developer time. And we, security people are constantly asking for developer time and that raises a lot of, uh, uh,

Matt Coles 4:11

We don't

Izar Tarandach 4:11

not expectations, but

Chris Romeo 4:13

It is a challenge.

Matt Coles 4:14

don't just ask for their time. We're not just asking for their time though, is there, right. I mean, security overall and, and we all do it to some

Izar Tarandach 4:22

Wait, I lost my audio.

Matt Coles 4:24

We're asking them to do something different. Right. We're asking them to expend resources to reveal information and that, and that isn't necessarily a bad thing, but we're also, you know, people love to say, oh, you're interrupting your, you're, you're affecting our velocity, right? So you're, you are impact impacting their ability to meet their needs and their requirements. Uh, and, and that's, and, and that is a very. Traditional, uh, originally I think a traditional view in the way that security operated. And now a more, um, what a word I'm looking for. It's not traditional, like it's completely not in my head. Um, uh, stereotypical, that's the word I'm looking for. It's a very stereotypical view of, of security. Oh, they're always roadblocks. Right. We've, we've gotten a, I think I'm a not un, not a, not a, a not undeserved label of being roadblocks to progress.

Chris Romeo 5:30

Hmm,

Matt Coles 5:31

And I think we're getting better about that over time. And, and I say we collectively in the grand scheme of security, uh, professionals, um, But, but stereo, typically, I think we're, we're seen as, you know, speed, bumps, roadblocks, and, and otherwise nets on the process.

Chris Romeo 5:48

We talked in a previous episode about guardrails and paved roads as a, a thing that limits that friction because the paved roads are there to give the, the guidance and give the, the correct components, the IAM policies, the things that are hard for individual, uh, developers to have to try to figure out and come up with a secure configuration for when you give them that paved road. But when I think about the challenges that engineering has, for example, with security, if we go to the, to the, the, the executive or leadership level, I, I think that this is part of the challenge. This is part of the, part of the why. Part of why they don't. Always embrace everything that we want to do. Because if, for example, I'm running an engineering team that has, let's just say a hundred developers, let's just make a nice round number and you're telling me that you want to introduce this thing called threat modeling, and I ask you, well, what is that gonna, what's that? How, how much effort are my people gonna have to put in? Well, uh, if they each expend 800..., 8 hours a month, We think we can, we can really make a move on this whole threat modeling thing and get better at it as an organization. But then what I have to do as a leader though, is I have to go, okay, I have a hundred people times eight hours. How many person months is that? How many person years is that? Like that has a cost that has a gigantic cost. When you start talking about doing something across a hundred. Now imagine if you have a. Thousand developers or 10,000 developers. Now it's eight hours times 10,000 developers. That's people I have. You're basically asking me to take a classroom full of people and dedicate them to threat modeling for the whole year when I, when I do the aggregate kind of analysis. So I think that's part of the challenge.

Izar Tarandach 7:31

But here, here's the thing, once upon a time when I was, um, perhaps not so young, but much more naive, security professional. I would look at you and say, but I need the time because I'm going to give you A, B, C, D, E, and things are going to be better at, at the end of it. But now I, I, I can already look around and, and empathize with those people and see how they, they think and see what's behind the thought from their side, and understand that most times what I'm putting on the table is something that's very intangible. Against the very tangible thing that you just put of the 800 hours in there. Right. So it, it's not even, uh, the fact that I'm coming and asking something that's very expensive to them, it's that what I'm putting against that it's something that they can't value beforehand. It makes it very hard for them to achieve a, a, a, a positive decision.

Chris Romeo 8:31

and it's hard to see the return on investment.

Izar Tarandach 8:34

Exactly.

Chris Romeo 8:34

When you're telling me, you can tell me, Hey, for that 800 hours, you gonna, you're going to have, on average, you're gonna eliminate two security vol. Two security, let's just say bugs, that you'll have to, you'll have to expend time and effort on a, from a re, rework perspective. So we can do an equation, we can do some formulas. But myself, as an engineering leader, if I take all my security experience and I throw it away and say, I'm just gonna pretend I'm an engineering leader, and I just know engineering. Show me the money type of scenario. Like you can't prove to me that you're going, I'm gonna expend the 800 hours. That's a, that's a given, but you can't prove to me that I'm gonna get 1600 hours of relief fixing two bug. You know, two security bugs are gonna get eliminated from each of those developers world, which is actually gonna double. You're gonna invest eight hours, but they're gonna save 16 or 24 or 40, or whatever it's going to be.

Izar Tarandach 9:29

Because again, you come in and you're selling insurance, right? Unlike established insurance, we, we don't have enough data to point out to, to actuarial tables and say, these are the chances that I'm going to make your your word a better word if you do the things that I'm asking you.

Chris Romeo 9:45

Hmm.

Matt Coles 9:46

It's actually interesting that you say that you use that analogy of selling insurance, right? Like. Farmers or, or State Farm or whatever to, but think of who we're talking to. We're talking to home builders, contractors and carpenters. And, uh, to use this analogy, right, we're talking about trades. Uh, the people who, you know, do plumbing and electrical and whatnot, these are, these are the engineers we're talking to. They don't get homeowners insurance for the, for the homes. They build. So why do we talk insurance...

Chris Romeo 10:23

they get liability insurance.

Izar Tarandach 10:24

Yeah.

Matt Coles 10:24

get liability insurance

Chris Romeo 10:25

They carry liability insurance, which is the, I think is the, the connection to what Izar is saying. You're offering me as an engineering leader, liability insurance, saying I'm gonna get 16 hours you that we're predicting this event's gonna, you're gonna have some number of events and I'm telling you, I'm gonna save you twice as much time.

Matt Coles 10:41

but they also get data. They also get data sheets. They also get, um, uh, regulation, you know, uh, uh, Permits and regulations from, from towns that they have to follow. Right. We're not talking about... Go ahead, Izar.

Izar Tarandach 10:57

Just, just to strangle the analogy here,

Matt Coles 10:59

Yeah. Cuz that

Izar Tarandach 11:00

if, if I ask somebody to build me a house, I get a subcontractor who's going to do the electrical? That person does their work. I move into the house. The house catches fire, and the fire firefighters say, Hey, it started because the electrical was not up to code, and something just burned up. You wanna tell me that the, that person that put your electrical in place doesn't have any liability? They do.

Chris Romeo 11:29

Mm.

Izar Tarandach 11:29

So, there there is, uh, uh, the issue that I, I can choose to go for insurance or I can choose to sit on top of that person and make sure that everything that they they're doing is up to code. So we are the guys who are...

Chris Romeo 11:41

who are. That's why you have building inspectors, right? Like I've heard people use this house building analogy about software as well, and you have the building inspectors who are doing the checking. So, The security team effectively is doing the checks at the right points and the inspections to say, this thing's been built to whatever the code was, the requirements that we've declared that where they're gonna be,

Izar Tarandach 12:04

So do people hate building inspectors?

Chris Romeo 12:07

yes.

Izar Tarandach 12:09

I, I, I don't know. I'm asking cause I have no clue.

Chris Romeo 12:12

I, I think when you're a house builder, when you're a builder and you have, you, you have a, just like in security though, with home, and I'm not a home builder by the way, but you have a range of inspectors. You have inspectors that will hold you to the every letter of the law. Oh no. It says that you have to have 74 nails per foot. I counted 73 here. Fail. Like, does, does that one nail not being at 74 somehow reduce the structural integrity of the house? No. That's just a, so the inspector can be a stickler that. And, and we as the consumer, we as the people who are gonna move into the house, we want that inspector to be like, 73, you're gonna gimme 80 cuz you missed one. You're gonna give me more. We want the cons as the person who owns the house, we want it to be as safe as possible, but as the person who's building the house, You want a happy medium. You don't want the inspector who's, who's giving you a list of 5,000 items that are wrong with the house, including, um, there was an extra stone in the driveway that was sitting in the front yard. A pebble, uh, you can't have pebbles off the driveway. Pebbles have to be within two feet of the driveway.

Izar Tarandach 13:26

So what it, what, what I think you're telling me because in the middle I had to change my headphones and I missed half of that. Uh,

Chris Romeo 13:32

It was brilliant by the way.

Izar Tarandach 13:35

I, I'll, I'll hear it. I'll I'll, I'll hear it on the episode. I'll hear it on the

Matt Coles 13:38

was epic. Yeah.

Izar Tarandach 13:40

on the, um, impression that I got from the tail end there is that I would hate that person less if what they brought to the table was wait for it reasonable.

Chris Romeo 13:54

Yeah, somewhere in the middle between extreme and weak there is a reasonableness that sits in the middle, even in the home building process where, and, and a lot of inspectors are reasonable cuz they work with builders all the time and and whatnot.

Matt Coles 14:10

Yeah. Pragmatism. Pragmatism, is that other word? We should be, we, we could use there.

Izar Tarandach 14:15

So pragmatism, born from empathy and perspective taking

Matt Coles 14:21

And experience,

Izar Tarandach 14:23

and experience, meaning.

Matt Coles 14:24

so, right, because, because that home builder is going to know, well, 73 versus 74, you know, the building code is, is designed for the 120, 130, 140% scenario. And so that's okay. They can make that reasonable call if there's nothing else wrong or that there's nothing else of significant value of, of interest, right? They can make, be pragmatic about, about that.

Chris Romeo 14:54

Mm-hmm. Okay, so why else? We talked about resource. I think I, we can bucket this whole conversation in resource management is a reason why engineering or engineers dislike what they get from security. But there are, there are, there are plenty more buckets like I can think of. Not... improper tuning or fidelity of tool results. Is a reason that developers, and that's not a resource thing, that's just annoying cuz your tool generated 500 Jira tickets for me, and now I have to go click through 'em and try to resolve all of them.

Matt Coles 15:26

You know, at the fundamental root, I think this comes down to the eyes over the shoulder, the over the oversight, the somebody else is paying is somebody else. There's a measure of control, but there's also a measure of somebody else is telling me how to do my job.

Izar Tarandach 15:44

Yeah, yeah, yeah, yeah, yeah. And, and that connects to what Brooke always says, that, uh, nobody likes to be told that their baby is ugly. So sometimes we go threat modeling, but we end up criticizing a design when we shouldn't be.

Chris Romeo 16:02

Mm-hmm.

Matt Coles 16:04

Well, it needs to happen. Just not a critical eye to the design needs to, needs to be placed, but we shouldn't be criticizing it cuz then it becomes personal for the developers.

Chris Romeo 16:15

Yeah, we're not critic. We always, and when I think of like something we could add to the Threat Modeling Manifesto, not that we could add anything to something that was so perfect, but there could be an angle there about, you know, respecting the person, respecting the engineer. I'm trying, I can't word it correctly, but basically respecting the engineer, attacking the design so that the person knows, Hey, I don't think you're a bad engineer. This is just what the process is, is we need to think about what could go wrong. I'm not saying this is even gonna happen. I think you're an excellent engineer, but some people could be, could be, could take that very, it could. It could be a rough receipt for them. Like you're, so you're telling me I'm a bad engineer, Izar. You're telling me I suck at coding.

Izar Tarandach 16:55

The, the, corollary, the, the corollary of the, the discussion with Brooke on that was that when we come in, it's not that we, we, we are telling you that your baby is ugly. We have to make it clear from the beginning that we are here just to make sure that, that your baby survives what it looks like is your problem. Which crib you're going to put it on. It's your thing. No, we, we we're not even a pediatrician. We're, we're the, the, the, the,

Matt Coles 17:24

The neonatal nurse. The neonatal nurse.

Chris Romeo 17:26

Guardians of the Crib. Guardians of the Galaxy, Guardians of the Crib. You can choose.

Izar Tarandach 17:30

No, the guardrails of the crib.

Chris Romeo 17:31

Ahh Guardrails of the crib. Oh, that's nice. That's a good, that's a nice visual illustration.

Izar Tarandach 17:39

A a and that sort of brings us back to, to what you said of the discussion that we had on guard, guard rails and safeguards. I, I don't think that developers like those specifically because it reduces the friction. I think that it reduces the friction because by putting those in place, we sort of disappear in the background. We security professionals just disappear. We, we. We infuse the development practices with those guardrails and safeguards, and that let, lets us take a step back and just look at how they're being used, rather than, are you doing the right thing? Are you doing the right thing? Are you doing the right thing? And now and now. Are you doing the right thing now? And now.

Chris Romeo 18:20

That's, developer enablement though. Like that's really the end game to what we do. If we can, if we can integrate AppSec slash security slash whatever does secure by design slash threat modeling slash whatever we wanna call it, if we can integrate that into developer workflows, processes, and tools instead of being something different like we talked about before in the previous episode as well. That's how you really eliminate the friction, cuz then this is just how we build software. It's not, we're not doing security things. We're just building software in a flow.

Izar Tarandach 18:53

But is that the end game? Cause I, I, I love how we, we. We build arches here without even knowing things that we, we talked about in previous, uh, episodes. At the end of the day, the end game is not putting those safeguards and guardrails in place and giving people tools that work and need less tuning or give less false positives. I think that at the end of the day, that the end game is for for for developers to have their own security mindedness. And realize that those are tools of their job, tools of, of their trade. Right?

Matt Coles 19:30

Similar to quality.

Izar Tarandach 19:32

Exactly. So

Chris Romeo 19:33

I mean, I think that's, I think that's a pie in the sky view. I think that's the

Izar Tarandach 19:37

it is aspirational, of course. Yeah.

Matt Coles 19:39

I think we need to work. We need to work towards it though. We,

Chris Romeo 19:41

but what I'm saying is developer. Developer enablement. Putting those functions and features and things into the existing tools like GitHub and GitLab, for example, are going through this, this, this takeover of the market where you can have SAST now from GitHub. That's in GitHub. You can have SCA, you can have all these things. They've put those where the develop they, they've integrated them into the development tools, so the developers don't even see something different.

Izar Tarandach 20:09

Oh,

Chris Romeo 20:09

so that, what I'm saying is that's a gateway to what your inspiration is. You want developers to just care when you can get there, but this, one of the steps along the way is to help them see is to take away the friction of other tools and other things that are outside of their normal flow.

Izar Tarandach 20:22

So let, let's go there for a second. Yeah, GitHub is giving them that frictionless thing, but when they receive the notice from, from the tool saying, there is something wrong here. Is their reaction, "Yay! The tool found something wrong that I can go and fix?" Or is it, "Oh God, one more thing that I have to stop now and go deal with because of this damn tool." That, that, that's the last step of the, of the thing here that, that's the aspirational part.

Chris Romeo 20:59

It's an 80/20 breakdown. By the way. I think 80% of the people are like, "Oh. Again?" There's 20% of people out there that are like, okay. I think, I think we're, I think there's 20% of the people that are, that are with us. Maybe I'm being too, too excited or whatever in this, but I feel like there is a movement of people that get what we've been teaching for 15 years or 20 years here, and they understand maybe it's because they've had a, a data breach or a gigantic vulnerability as a result of a mistake they made. And so now they've just got it. They're like, I, I felt the pain. I get it. I gotta deal with these things. I don't know. I don't know where it lands.

Izar Tarandach 21:37

Yes. But to go back to the question of today, do those people stop hating security professionals? Or do they stop seeing them as extra work because now they see how those things are actually necessary?

Matt Coles 21:58

So I wanna, I wanna add, throw out a question to you guys with respect to developer enable. If we look at developer development enablement as sort as a, a phase where security blends into the background, right? Where we become, where we create things or specify things or guide people to adopt tools and other things, training and whatnot, that helps them do their job and include security in it. And that's not necessarily a bad thing. I'm, I'm not su not in this question, I'm not suggesting anywhere that it's bad to be on that path, but do you think that there's a place, the, the way that, the way I like to think about how I don't piss off developers that I talk to is to be a. To be a trusted partner. That's a real, that's not the, that's a very, uh, overused way of saying it. I think. Um, you know, I enjoy sitting with the engineering team and becoming a part of their group and working with them closely. Yes, it takes a lot more time. It doesn't scale very well, but it helps that we can talk about engineering problems together and introduce security into that without it being, oh, you have to do this because this policy says X, Y, and Z, and you know, I'm here to enforce it. No, I'm, I'm an ar, I'm an an architect, you know, I'm bringing the security view, the, the, the voice of the customer. We can debate as engineers. Right? And, and you gain trust that way.

Chris Romeo 23:46

It's a scalability problem though.

Izar Tarandach 23:48

It's not, I, I,

Chris Romeo 23:49

do I, do that with 10,000 developers?

Matt Coles 23:51

you have to, or maybe. Maybe Do you have

Chris Romeo 23:53

security person.

Matt Coles 23:54

but.

Izar Tarandach 23:55

I don't think.

Chris Romeo 23:56

every 25 maybe. So now I basically need my 10,000 developers. I need four to 140 to a thousand, 40 times. I need 400 secure. I need 400 Matt Coles, which I would take that, don't get me wrong, I would have an organization of 400 Matt Coles

Matt Coles 24:12

We, we had, we had the, we had this challenge, uh, uh, two decades ago with how many quality engineers do you have per developer? Per, per, how many quality? How many developers do you have per quality engineer?

Chris Romeo 24:23

Yeah. And we, I mean, we never solved it. I don't think we ever

Matt Coles 24:26

to one. Something like that.

Chris Romeo 24:27

Yeah. And, and the same thing's happening here. Like that's, that's a, i, I, I think that's a, a, a noble goal, but I have yet to find anybody who can resource to it, because you can't resource to it with 10 people.

Izar Tarandach 24:38

But this is not going to solve the problem as This, this doesn't solve it. This, this just moves the, this just reduces the, the, the, the blessed radius of the problem.

Matt Coles 24:50

does it solve, doesn't it solve the, how do we not look like jerks to, to developers?

Izar Tarandach 24:55

Yes.

Matt Coles 24:56

Okay, so we've ans we we can solve it. It's a way, It's

Izar Tarandach 25:00

no, no, no, no, no, no. Sorry. You said, does it solve or does it not solve? I, I'd say it doesn't, does not solve, and the the reason is, is very simple. Once upon a time you were developing something, you would open a terminal, open vi write code, compile, push. Nowadays things have reached a level of spec spec specialization both for the developer and for the, the security person. That creates a, a sense of otherness that for each other we, we, we are always going to be separate species, but with some common DNA and that otherness. I, I, I, I think it, it, it just, it just, it just makes the, the, the, the whole dialogue not more complicated, but more heavy. We, we, we are pulling in different directions.

Chris Romeo 25:59

See, I don't see that I, I see it in Matt now. I don't think Matt's model works from a scalability perspective, but I think it does work from a building trust and security is no longer this faceless entity that's throwing work at us because Matt's there with that team of 20 engineers and he's like, Hey, I understand this sucks that we just, we just did this tool to spit this out. I'm gonna roll up my sleeves now and let's see if we can get this going. Like, you're leading, you're not. You're not just saying, Hey, good luck folks. I'm gonna go home. I know it's 3:00 PM I'm heading home for the weekend. Enjoy. You're there rolling up your sleeves. You're with them, and they're like, Hey, Matt is part of this team. He's with us. He's, he's leading from the front. He's helping us solve the problems that he has control over can or can influence. And so I think, I think I'm with Matt on this one. I think they do end up trusting you. They, they, they lose their dislike because you're one of, you're with them, you're part of their team, you're embedded on the team.

Matt Coles 26:52

Mm-hmm.

Izar Tarandach 26:52

But no, because the problems that we solve as security people are removed from the problems that they solve as development people. Their volume is much bigger. We appear once in a blue moon to deal with a finding or to deal with threat modeling or to deal with this thing or death thing.

Chris Romeo 27:14

That's not the model we're talking here.

Matt Coles 27:15

It's not the model

Chris Romeo 27:16

In Matt's model, Matt is with the team.

Izar Tarandach 27:18

Yeah. With the team that, that's what I'm saying.

Chris Romeo 27:20

He sits in there where their cubes are. He has a cube there. He hangs out with them all day.

Izar Tarandach 27:24

And most of the time he has his thumbs and they go circling until something shows up.

Matt Coles 27:29

No, no, no, no, no. So

Izar Tarandach 27:31

Unless you're telling me that you become a developer, that just happens to be a security expert as well.

Chris Romeo 27:35

Well, I think you do some development in that world though. You do.

Izar Tarandach 27:38

I, believe me, I don't want my code going into production.

Chris Romeo 27:42

Oh, mine's not allowed. It's been banned by the whatever code institute of the world.

Izar Tarandach 27:45

So what am I doing? I'm, I'm reducing the quality of the, the team because either I embed as a hundred percent security person, or I embed as a poor engineer that knows security. If I am a hundred percent security person, I'm sorry. Even in my most self, self-aggrandizing dreams, there isn't a hundred percent of the time security issues to be addressed. So I am basically idling there until something happens. It's like I'm going to bring a firefighter.

Chris Romeo 28:17

You're leading the threat models. No. You're, you're, there's no way You're idling, you're leading the threat model process.

Matt Coles 28:22

Participating in architectural reviews...

Chris Romeo 28:23

You're doing code reviews you're doing code reviews of PRs that are coming through.

Matt Coles 28:27

Well, and in fact, in fact, I would, let me just, let me just throw out a, a, an example, right. Uh, in a previous life, uh, working with, with a comp, with, you know, engineers at this is, was the way it worked and it was a small company, made it a little bit easier, right? I sat in design reviews, in architecture reviews when they were talking about how to implement something. I was present because with the ensuring skills that I, that I could bring to the table, in addition to the security knowledge, I could act as another pair of eyes as a peer, that built trust. So while I wasn't writing code, I was influencing the design so that we didn't have to do security later. We can influence, we can influence the thing at the ground when they're defining it or when they're implementing it, or when they're deploying it, or, Hey, so-and-so customer just called and they have this complaint or issue. We can do the translation.

Izar Tarandach 29:27

But matt, that, that,

Matt Coles 29:29

yeah, I know that doesn't scale.

Izar Tarandach 29:30

no, it's not that it doesn't scale. That's exactly my point. You were able to do it back then because I'm, I'm willing to bet The stack that was used for that specific application and the deployment model and all that good stuff was way more contained than what we have today.

Matt Coles 29:47

Uh, I think that there's still a ton of companies out there, small companies out there, especially that's, that still happens.

Izar Tarandach 29:55

Yes, and those companies, the good ones do have security staff on board. Many of them still see it as a cost sink and say later on when we get bigger. But today, in the embedding model of the, the, the norm, not normal of the, the 80% of companies out there, my point is that to bring a security person in, That also has all the knowledge of the stack to be considered a peer by software engineers. You, you, you're looking for a a a, a four-leafed, uh, uh, four-leafed unicorn.

Chris Romeo 30:32

People call 'em unicorn devs. The DevSecOps unicorn idea is you have someone who knows Dev, Sec and Ops all together, and they're unicorns cuz you don't see them very often.

Izar Tarandach 30:43

And basically what

Chris Romeo 30:43

Everyone says they saw one, but no one ever can prove it.

Izar Tarandach 30:45

And what, what you end up is with a rhino that somebody painted white.

Chris Romeo 30:50

Alright.

Izar Tarandach 30:51

Okay. And you know what? I'm gonna put my hand up and say I'm one of those. I mean, I can talk technicalities with engineers all day long, but I know that I am not at their level. At the same time, I don't expect them to be at the level of expertise that I bring in the security side, right.

Matt Coles 31:11

That makes sense. Yes, I would agree with that and, and I agree. I, you know, I'm not an expert coder in many languages that they use, but I don't think you have to be.

Chris Romeo 31:23

Yeah, I agree.

Izar Tarandach 31:24

Because you are looking at the principles. But the problem is that once you agree on the principles on the design and they go do their merry sprint dance, you will sit back and wait for something to pop up to happen. You will not be doing 24-7 threat models and code reviews and this and that and the other one.

Chris Romeo 31:44

Engineering teams turn over five to 10% of their staff a year. That's, that's what people have always tell me that, you know, I, cuz I've heard that argument made in different contexts that like you're going to, and for example, in the training space, you're going to train everybody and there's not gonna be any more people to do it. Well, every year, 5-10% of people in your engineering team turnover. So there's always new people who are coming in cold. They don't have a perspective on, on educa or uh, security. They haven't so, I don't think, I mean this is, this is a weird place for me cuz I'm literally arguing both sides of the argument. I actually don't think this works from a scalability perspective, but I do think it works from an expertise and a trust perspective. So I'm in a really weird spot.

Izar Tarandach 32:26

I I'm

Chris Romeo 32:26

I should just be quiet.

Izar Tarandach 32:27

I, I'm not even looking at the...

Matt Coles 32:29

We're not talking about whether it solves the security problem, right? We're talking about whether or not there's a trust issue, and with that, this approach works to solve that trust issue.

Izar Tarandach 32:41

Look at at, at the end of the day, the trust. Okay? The trust. I agree with you completely because all of a sudden you have, okay, let, let's put things aside for a second. The scale scaling, the three of us agree, this doesn't scale. Great. So let's look at a hypothetical, small company, small team, but with the resources to have security staff at hand. So I agree with Matt, the, the, the trust issue gets solved because all of a sudden you have a security person at hand. So for the developer team to now say, whenever I have a security problem, I have a person that I can just turn 90 degrees and talk to them directly. That definitely solves the trust problem. What I'm saying is that very quickly what went up is that security person gets underutilized. And all of a sudden as a solution to that, because people in this line of work don't don't like to be bored or bad things happen when they get bored.

Matt Coles 33:42

All right.

Izar Tarandach 33:43

All of a sudden these people find themselves writing code for the product, right? Because you can only write so much security engineering until stuff starts to work and you're happy.

Chris Romeo 34:00

Doesn't that spur you on it as a security person though, if you're in that role, to expand your knowledge? Because it certainly did for me. So I've been in that role before where I was, I was kind of at the center of, of building something and I had brought my security knowledge and expertise, but I didn't truly understand the language and framework and everything. So I went and learned it because I was like, I wanna be more effective in what I'm doing. And so, yes, it took me a little bit of time. I wasn't, I didn't come in ready to solve problems, but over a period of three to six months, I got to the point where I understood that framework. I understood the language. It was Ruby on Rails. I'm like, I, I understand how it works. I understand what the challenges are. It made me a better security person. Cause now I understood it from the inside out. So I wasn't ready on day one, but by day 180 I was. You still didn't want my code going into production, but I had, I had taken steps to better myself cause I wanted to be more effective with my my peers.

Izar Tarandach 35:01

But I think that there's a reason why most AppSec people, and and I make a point here to say, just the AppSec, not definitely not all parts of security, why they tend to be bit more generalists than deep dive into issues, right? Because the technology out there moves really, really, really fast. And if we spend the time to go deep into something, we're going to miss the other 180 things that happened meanwhile. I I, I just don't think that that's, that that metamorphosis turns the security person into something that the developers automatically like more.

Matt Coles 35:48

Not automatically.

Izar Tarandach 35:49

I mean, I, I, I can't imagine, I, I can imagine that security person in a sprint, everybody's talking about something and they raise their hands and say, we cannot ship because we didn't do the, uh, authentication authorization right. And everybody looks

Chris Romeo 36:02

Hopefully you told them at the beginning.

Izar Tarandach 36:03

No. And everybody looks at them.

Chris Romeo 36:04

Hopefully you're telling them at the beginning.

Izar Tarandach 36:05

Perhaps

Matt Coles 36:06

hopefully you don't

Izar Tarandach 36:07

perhaps they were

Matt Coles 36:08

as, we can't ship that. That defeats the purpose.

Izar Tarandach 36:09

No, perhaps they were telling all, all the way through and they were being told. MVP. We have to put things out. We have to prove that it works. Blah, blah, blah, blah, blah, blah. Bolt on. And then everybody looks at, "Dammit Carl." Really?

Matt Coles 36:23

Well, so remember this word, remember that word, pragmatism, right? You in this model. In this model, there are times when you have to be comfortable. It's in part a trust thing, and it goes both. It's sort of a trust in two aspects, right? There's, there's getting developer trust and help and having them recognize you as a peer. Sometimes that means smoothing over the edges when it comes to the way that com, that compliance, hard compliance things, right? Can you not ship? Because you don't have authentication. Well, maybe MVP, sorry,

Izar Tarandach 37:08

That, that's how we sound for, for developers by the way.

Chris Romeo 37:13

Great illustration. That's what we sound like is just a barking dog always nipping at their heels.

Matt Coles 37:19

so maybe hold on.

Izar Tarandach 37:24

But I, I think that at the end of the day here, uh, Matt makes an excellent point. This makes people trust you more. But at the end of the day, the, the, the question that we are battling with today, just because they trust you, do they like you more? Sorry. Just because they trust you. Don't they hate you?

Matt Coles 37:43

They hate you less Certainly.

Chris Romeo 37:45

They hate you less. They like you more cuz you're, you're humanizing your function. You're not a, you're not a, a box off to the side who's, who's sending things across, throwing it over the wall, and we don't even know who's over there, but they're just telling us our stuff is terrible and we can't, we can't move forward.

Izar Tarandach 38:01

So you, you you

Chris Romeo 38:02

it's, Matt said, our stuff is not great, but he's helping us figure out how to fix it,

Matt Coles 38:07

right. And realizing that and realizing that MVP may me, may mean good enough and not great. Right?

Izar Tarandach 38:14

You, you think that somebody's saying he's a pain in the ass or somebody's saying he's a pain in the ass, but he's right. That,

Chris Romeo 38:22

But he's our pain. painted

Matt Coles 38:24

Right,

Izar Tarandach 38:25

that, that involves

Matt Coles 38:26

it goes because the other, because it isn't just an inward facing thing, it's an outward facing thing, right? You're there to defend and translate from other teams, right? So when the security team or the compliance teams, your C, whoever comes along and says, Hey, you're not meeting this. The standards like, well, But they're doing the best they can. Now you become the voice when you're talking to the developers. You're voice of the customer. When you're talking to others, your voice of the developer, you can provide that layer. It's not

Chris Romeo 38:52

I think, uh, I think, I think Matt and I have have made our point here. We,

Izar Tarandach 38:55

so let, let's, yeah, let, let, let's say that you, that you manage to convince me fine. Okay, cool. Now we are talking just the technical aspects of the thing. Can we step back and talk a bit about the soft skills of security people?

Matt Coles 39:12

All of this requires soft skills. All

Izar Tarandach 39:15

do, to do it right? But if you don't, but if you don't, it's still effective because you're talking tech technicalities. You're talking. So something that you can, you can point at a line of code and say, this is not a line of code that should be here.

Matt Coles 39:27

You

Izar Tarandach 39:27

if you do that nicely or if you, if you, if you're a pain in the ass about it.

Chris Romeo 39:31

it's, I mean, I've come to reali, I came to realize, uh, it's probably been 10 years, eh, maybe not that long ago since I realized it. But, uh, I started talking and I know Izar, you talk about this as well, this whole idea of developer empathy. As security people. If we don't have empathy for the developers, then we really are not very effective and we're not reasonable. Just to use another term that we've been throwing around, it's not reasonable to, to be unempathetic of our developer partners in this process. And when I think developer empathy, it means I should realize the pain that my tooling inflicts on people's job operations, what they have to do on a day-to-day basis. I should understand, how that impacts them, how it impacts the team, the resourcing. But I should also. It's, so, empathy is one thing. Soft skills is another thing. And, and soft skills is more of a thing about how are we influencing, how are we, how are we leading, how are we influencing without starting a fight but instead collaborating and making an argument. But for me it comes back to empathy is always at the core of that. Like if I don't have empathy for my development teams, I'm, I'm never really gonna get very far cause I'm gonna be stuck in all the old traps that we've had for the last 25 years.

Izar Tarandach 40:50

you

Matt Coles 40:50

there are ways to solve the scaling problem, by the way, there are things you can do. no, no, not perfectly. Not perfectly. Sorry. You can help with the scaling problem, right? Things like making it a rotation, right? So maybe you're not with them full-time, or maybe with them full-time for a period, a period of time.

Chris Romeo 41:09

Yeah, but then you don't, you don't have the trust

Izar Tarandach 41:11

and then you you introduce,

Matt Coles 41:12

you, you gain, you gain trust over time with that, with that team. And you, as you're rolling off of that team to the next one, you have spent time training the trainer, working with the champion and getting the champion confident in their role, right? So, so you become a member of the team, a connected member of the team up here. You gain trust, you get invited to things, but you're there to help with the exp... Explicitly, you're there to support them, enable them support and enable them, really. Right. So, and, and again, none of this is perfect. It doesn't solve the security problem. It solves

Chris Romeo 41:50

But that's a coaching, what you just described there is a coaching program, not a, not a developer. Embedding not, not embedding a security person.

Matt Coles 41:58

I think they can be done together. I know they can

Chris Romeo 42:01

drop in, parachute in, and then they go away

Matt Coles 42:04

Not that's, that's not what this is, that's not the type of program that I'm talking about. the, the, the

Chris Romeo 42:09

I want, I want to, we're almost outta time here and I want to get, I wanna make sure we capture Izar's empathy thoughts. Cause I know he's, he's talked about this a lot. We can, we can have another episode where we argue about how Matt's idea was not scalable and there's no way that it possibly,

Matt Coles 42:23

I'm not gonna argue against that. I just say,

Chris Romeo 42:25

Uh,

Izar Tarandach 42:26

No, I, I,

Chris Romeo 42:26

be a short episode. But Izar, what do you think on empathy? What, what, what do you talk about with people?

Izar Tarandach 42:31

I, I, think that I, I see the point of what, what Matt's saying, right? And, and, and I, I agree to an extent. To go back to the empathy, I think that it was very interesting that you pulled the empathy thread to the side of us as security people, looking at the tools and things that we give the developer and feeling their pain on what they're, they're suffering in their hand. But another thing that we always joke about is that, uh, you measure security code reviews by WTF per second, and. One thing that we miss is that sometimes those WTF, they, they have a reason. Developers live in, in a constrained world that, that they're not free to do whatever it is that they want, and sometimes it may happen that the things that we ask them to do lie beyond the constraints that they have for a number of reasons. And, uh, uh, uh, an important part of our empathy is, is, is understanding how they live in that constrained world and how what we asking can be the, the straw that breaks the camel's back.

Chris Romeo 43:38

Hmm. That's a good thing to remember as well. That is, I, I, I, I can say I haven't thought about empathy in that particular vein that you described, but I think it's a powerful. Thing that I should be, I should be thinking about. I am I asking them to do something that's beyond what they can do. So they can't e they can't make me happy and they can't make the rest of the team happy by fixing whatever the issue was or whatever. Like they're stuck in a, a vortex almost.

Izar Tarandach 44:02

And, and that leads to another thing, something that I, I learned from my wife that we use that word empathy, but it's almost like risk. We, we use the word, but we mean something else. People who have empathy for each other run the risk And that's not the one people who, who, who, who, who feel empathy for each other, they. Run the risk of falling into the trap of commiserating together. So they realize that there is a limit to what they can do. And they, they dwell in that, oh, well that's the suck. That, that, that's, that's what we have to do. Right. And I think that what we are actually talking about here is perspective taking. It's that getting to their shoes and, and look at a word from that angle rather than, Feeling bad because they are under constraints or, or something like that.

Chris Romeo 44:56

Do people really commiserate

Izar Tarandach 44:57

Oh, I commiserate all the time. I commiserate all the time,

Matt Coles 45:02

Yeah.

Chris Romeo 45:02

I'm

Izar Tarandach 45:03

with myself.

Chris Romeo 45:06

it really commiserating? If you're with yourself though,

Izar Tarandach 45:08

Well, it depends.

Matt Coles 45:08

schizophrenic, maybe.

Izar Tarandach 45:09

it depends how many voices you have in your head.

Chris Romeo 45:12

That's true. That's true.

Matt Coles 45:14

Or, you know, peyote or LSD whatever you...

Chris Romeo 45:18

Let's circle back around and let me ask the question one more time cuz I want and then we gotta, we have to tie it to the sound of silence somehow, because that's what I offered in the beginning. But the question we asked was how to be a security person. How do you be a security person that engineers don't hate? And so just to recap, we talked about. A number of different things, but if we work our way backward, we talked about empathy and soft skills and influence and embedding developers within that environment. Um, resource management. We talked about, uh, the over the shoulder th lick. Look at people, you know, people don't want to be looked over the shoulder. They don't, they don't like that, I guess. But final answer here, 30 seconds or left for less for each of you. How would you, how would you, what would be your, your summary of how to be a security person that engineers don't hate based on your own unique experience? Matt, first.

Matt Coles 46:16

I, I, I, I have nothing else to add. Uh, really, it, it's about, not, it, it's about being part of their world. I, I, I don't think there's anything more than that.

Chris Romeo 46:30

Okay. So you did have more to add cuz that was more so just fyi. That was a good summary though, of being part of their world. That's a good summary. You said it differently than you did before, but it gives you another view of, oh, okay. I'm part of their world, I'm part of their existence and how they're doing their job. Izar, what about you? What's your, your

Izar Tarandach 46:47

So a, a, as we speak, one of the voices in my head is going back and reminding me that I was there when I saw Matt do what he described when I saw him embedding with the team, and I saw the results and I saw how much respect he got from that team. So I'll, I'll have to go back and say, yes, Matt is right. That is extremely important. Then when we extrapolate to the problem, if it doesn't scale, then I think that if we take into consideration everything that we talked about here, empathy, uh, perspective taking and, and all that good stuff, and embedding, add all that together. And I think that the one-liner that I come up with is get closer to them. Don't, don't remove yourself from the picture. Insert yourself into the picture that they are pen painting. Understand that it's not your picture. You, you are just background. You're just there to make the picture look better. Well, in some cases, but, uh,

Chris Romeo 47:42

and I think, I mean, that is. The thing I take away from this as well, all of the things you said, you know, the and, and the different areas that we talked about, but it is a scalability problem and, and maybe there is things we can do. Maybe there's there, there are things we can do to try to embed security with less than a 25-to-one ratio of devs to security people, because I mean, we're all gonna agree. We've never seen a security organization that was allowed to grow and build a embedding program. That was at those levels with, you know, 10,000 engineers and 400 security people that were just doing embedded efforts inside of the team. And, and we don't see that, I don't see that being a reality, especially as

Matt Coles 48:28

we sort of meet halfway.

Chris Romeo 48:29

less.

Matt Coles 48:30

We go about halfway with security champions, right? Because they're developers who gain security experience, not the other way

Chris Romeo 48:35

That's a good, I'm gonna, I'm gonna, If we're gonna, we're gonna press pause on that and we're gonna carry that forward to a future conversation of the difference between embedding a security person versus a security champion. And what are the things you get from each of those.

Izar Tarandach 48:50

Oh, we're going to talk about security champions. Yay.

Chris Romeo 48:52

yeah, yeah, yeah.

Izar Tarandach 48:54

Well, we had to

Chris Romeo 48:55

Pandora's box though, of opening that because,

Matt Coles 48:57

Yeah.

Chris Romeo 48:59

I almost wanna start making my argument. I'm not going to, I'm gonna say thank you for joining us here on the security table, talking about how to be a security person that engineers don't dislike. Hopefully that describes you. If it doesn't, hopefully you can apply some of these things we talked about so that people will like you just 1% more per day. Thanks everybody.