The Security Table
The Security Table is four cybersecurity industry veterans from diverse backgrounds discussing how to build secure software and all the issues that arise!
The Security Table
Threat Modeling Capabilities
This week around the Security Table Matt, Izar and Chris discuss the recently-published Threat Modeling Capabilities document. They explore how capabilities serve as measurable goals that organizations either possess or lack, contrasting the binary nature of capabilities with the continuum of maturity. The team shares insights on the careful definition and measurement of each capability, highlighting the creative debates and diverse perspectives that enriched the document.
They also emphasize the collaborative effort behind the document's creation. The process mirrors the successful teamwork from the Threat Modeling Manifesto, showcasing the enjoyment and effectiveness of their work together.
Finally, the team reflects on their journey from the project's start to the release of the Threat Modeling Capabilities document. They share personal stories and the collaborative spirit that led to the project's success, inviting feedback from the community to refine and improve the document further.
Links
Threat Modeling Manifesto: https://www.threatmodelingmanifesto.org/
Threat Modeling Capabilities: https://www.threatmodelingmanifesto.org/capabilities/
FOLLOW OUR SOCIAL MEDIA:
➜Twitter: @SecTablePodcast
➜LinkedIn: The Security Table Podcast
➜YouTube: The Security Table YouTube Channel
Thanks for Listening!
HACK THE WOOOOORLD! Hehehe, PEACE BE WITH YOU!
Chris Romeo:So welcome to the Hack the Planet, a new podcast that is a, uh, it's a rewatch of, well, that's maturity, that's, yeah, that's, now, wait, Matt, were you trying to say, like, that's real mature, you? Okay, that's real mature.
Izar Tarandach:As I said, I refuse to have anything to do with anything that's measured in maturity.
Chris Romeo:Now we're talking about like, in your entire life, just a technology.
Matt Coles:Who
Chris Romeo:There's another t shirt for you. PPFL. Pan for life.
Izar Tarandach:By the way, I have approval. I have approval for the LLM t shirt. So, I may go there.
Chris Romeo:from?
Izar Tarandach:Well, you know, when you say I identify as You better ask people around.
Chris Romeo:okay. I see.
Matt Coles:owns LLM as a copyright?
Izar Tarandach:Nah, not me.
Matt Coles:Well, you asked somebody.
Izar Tarandach:Oh no, somebody who understands, like, somebody who is more. aware of these things than I am,
Matt Coles:Uh,
Izar Tarandach:which could be said to anybody, but no, I went to a,
Chris Romeo:You're quite aware of. You seem to be vaguely aware of your surroundings. I've noticed that about you.
Izar Tarandach:I'm surrounded,
Matt Coles:You were surrounded.
Chris Romeo:Or
Matt Coles:behind you, the fish behind you is listening.
Izar Tarandach:I got Kermit. Yeah.
Chris Romeo:about something here, maybe? Or is this just the random show versus the security
Izar Tarandach:No, wait, today is celebration day.
Chris Romeo:it is?
Izar Tarandach:It is.
Chris Romeo:we need a celebration, so we have, uh, Ah, Izar is celebrating with fireworks, for those on the
Izar Tarandach:Now that I learned this thing, that's done.
Chris Romeo:Yeah, it's like a, it's like giving a small child a new toy that makes noise. is effectively Izar and Reactions, uh, a new feature that, uh, is the, the Canberra and Mac OS do for you. But yeah, we are here on the security table to celebrate what I'll define as a momentous occasion. So the three of us with another twelve people, I had to go count. in the document itself to make sure I didn't miss anybody, but three of us plus 12 other people. Uh, most of the folks were from the original Threat Modeling Manifesto team. Uh, we are proud and happy to announce that we have released another document. And this time we have released something called Threat Modeling Capabilities. And so we want to use our time in the show today to introduce you to This new project that we've just released, uh, two days ago, uh, from the time we're recording this, uh, to fill you in on what it is, but then also let's get a little taste of how it was put together, uh, by way of, uh, perhaps a little bit of discussion and debate about the, the merits of maturity over capability. But I, I, I I'm getting there too soon. So Matt set it up for us as far as like. What, what's the capability? Like, what are we even talking about when we say threat modeling capabilities?
Matt Coles:So, uh, in a nutshell, a capability is something that you do or have In your, in this case, in your program, your threat modeling program. So it's something that you do or do not do, as John, Jonathan would like to say. Uh, and so, uh, a capability is, um, and in this case, we're talking about capabilities, things that Uh, help to improve or make most efficient your threat modeling process and, uh, and separately, of course, we could talk about maturity, but, but that's obviously a separate conversation around how well, or how frequent, or how consistently you do that thing. So, uh, a capability really is. Uh, something that you achieve, an objective, a goal, something that you can measure, uh, and say you do or do not have or do not, do or do not do, and, uh, and that's, that's in a nutshell where we're at from this, from this document.
Chris Romeo:Izar, why don't you set up for those listening, how this document came together. Talk about the team and the process and how, how we work together to create this thing.
Izar Tarandach:So we, we tried to emulate what we did with the manifesto, right? So the team already knows that, uh, not only we work well together, but we enjoy working together. I think that both things are actually connected. Something to look at. But, uh, So we got together the same way, uh, once a week, an hour, and a lot of offline work over Google Docs. And we managed to I think that's the first big discussion that we had was the capability slash maturity: What are we setting up to do? And I think that we came to the conclusion that I, if I remember correctly, and if not, uh, hear it soon in an AppSec, uh, uh, podcast of your choice, but, uh, I think that we, we got to sort of the idea that the capability was more of a, uh, a zero or one. You either have it or you don't have it. And, uh, a maturity was a continuum. That thing that you have or don't have. The maturity would reflect how far along you are in that journey. So we decided that it made more sense to actually, uh, define what the capabilities are and only at a later date. Dun dun dun dun. Uh, going, go the, the maturity way. And, uh, uh, we got to divide all the capabilities into seven, uh, process areas, which are strategy, education, creating threat models, acting on threat models, communications, measurement, and program management. And from here you can already see that what we tried to put together was, uh, not only the capabilities that you may need to do a proper threat model, but we also looked at the program side of things. Many times people know how to do individual threat models, and that's great, and that's awesome, but how do you translate that to a whole organization? How do you build a whole program around it, right? What does the organization has to have in place? And, uh, it's hard to talk about those things without, again, using the word maturity. But, uh, how far along the journey does the security, uh, the organization has to be? in order to be able to both create, maintain, and extract value from threat models? And I think that's the most important thing here, is that we went exactly that way. How do you use these capabilities? How having or not having these capabilities influences you being able to extract value, and to nurture, and to grow your threat...threat modeling program?
Chris Romeo:So I wanna, I wanna Challenge you both here to, uh, and I'm going to go first, so you'll have time to think about this. I'm not just going to throw this question at you just randomly, but I want each of you to share. What do you, what was one of the big, like the, the things that you took away? What's a story that you can share about this journey that began in May and ended yesterday with the release of the capabilities document? Um, it could be something along the journey, like something that happened or whatever. I know there was a lot of cool things that happened. I'll share one first to give you guys time to noodle on it here. Um, I would, I would say that the part of this process that, uh, that just really stuck with me, that one of the things I thought was so fun, we were We did a workshop where we had all 15 of us in a shared Miro board, and we were brainstorming kind of like the positive and negative sides, like what are things that could go wrong in a threat modeling program, and then what are the things that would be things that could go right. And so we were looking at it from two different perspectives, uh, but Fraser Scott led us through that, um, that exercise, and we literally had, like, Uh, Fraser would make an announcement and it would be like a five minutes of work and you'd watch all these cursors flying all over this little shared Miro screen, dropping boxes in and it was just a really, it was just a really fun way to brainstorm at such a high rate of speed and like in 90 minutes or something, we ended up with the guts of what became the capabilities that we released yesterday. Now, they changed, they modified. Yes, there was a lot of tweaking and whatnot that happened to them, but still the core of those things came out of that. That brainstorming exercise and I'm looking forward to trying to use that, that style of, of a workshop again in the future, because it was just such a powerful thing to get so many smart people putting together their, their best ideas down at a high rate of speed. So that was the one for me. Um, I gave you guys plenty of time to think up like your, your key moment of the experience. So Matt, I'm going to you next.
Matt Coles:Oh, go with Izar. He looks like he has something
Izar Tarandach:No, no, no, no, you first, you first, you first.
Matt Coles:No, no, no, that's okay. You can go
Chris Romeo:Listen, if you use it, he can't tell it. So whoever goes next is locking in another thing out of the story.
Izar Tarandach:so, uh, to me, I'm going to be very, uh, selfish here and point at something that's very near and dear to my heart and that I think that I pushed very hard to get into the document. It's the fact that each capability is either measurable or provable. You're not able to self attest and everything's fine because I said so. You actually have to put some stuff on the table to show that yes, you have or don't have that capability and where it goes. And I think that the fact that we managed to do it so concise, I mean, each capability is what? Tops a paragraph. Some of them are just one sentence. I think that that's so cool that we're not like writing Bibles over here. Oh,
Matt Coles:Yeah, absolutely. I think the number one key, I guess, standout thing that happened from my perspective is actually something that happened regularly throughout the conversations was the process of understanding that we went through. Yeah. Right? So, starting with the concepts. What do we need in a threat modeling program in order to be successful and efficient and effective? And taking that, approaching it, breaking it down into, okay, what is it, what do those represent? Parts. And then having the conversation across, uh, across all the members, um, who are, who are present on those various, in those various sessions. Um. To, um, think about, we obviously, we had folks who were on the consulting or, or, uh, um, academic side or, uh, practitioners of threat modelers or developers of tools, uh, as well as, um, folks who were, who were consuming it and, and working with business and engineering teams. And so those different perspectives being brought together to really understand the problem, to really understand, even if we had a capability that could be measured. A, was it valuable to have? And B, was it going to actually be done? Right? There's a, there's a notion that some things are probably worth doing, and some things are maybe not so worth doing, and so therefore, You know, things get, things get tabled or, or held back or, or just not, not put forward as, as, as of, it may be a value, but not in context. And so having those different opinions come together, and then once we started looking at those and refining them, doing things like, right, that's maturity. So making sure that we were not doing, that we're not baking in maturity into the capability so that we had a thing, set of things that can be done and could be measured without worrying about the how or the to what level. And so that was, that was just amazing that we were able to hold it through through the entire deliberations, um, and, and maintain that, uh, with all the different opinions that were, were involved in the fact that we could come together as a group and make and come to a conclusion that we could release was, was simply, uh, amazing.
Chris Romeo:And that speaks to, I guess, I'll bookend this, this portion of the podcast. I shared the first one, I'll share the last one. It just speaks to the quality of the people, these 15 people that we, we brought together here. Matt, you already touched on the fact that all of us are coming from different perspectives, different jobs, different, uh, threat modeling experiences over our careers. Some of our authors, such as yourselves, um, others are trainers, others are consultants and. and running programs inside of companies. But there was never a moment in the deliberations around the capabilities where any, where anybody. Raised their voice or yelled at anybody else. Like it was a, it was just a peaceful group of very smart people. And that's one of the things that I just love about working with this team is we can respectfully disagree with each other. And we did. Hundreds and hundreds and hundreds of times in the midst of this, we did not all, we did, there were many things that we did not agree on, but everybody very respectfully lays out their case, and then other people will start to add additional data to it, and then as a group, We kind of come to a decision and then we move forward and we say, okay, here's what we're going to do with it. Because we feel like, and there was a few times where I called for a vote because I'm like, I just don't know which way this is going. Like we need democratic process to determine what the, what the outcome of this particular issue is. Because a lot of times it was like somebody would make a case, somebody might make a counterpoint, people started to kind of fall into one. And it became very clear, like, okay, this is where the group is leading. A few times we had to go to the democratic process, but never did anybody yell, scream, carry on, grandstand, um, filibuster, all the words we can use that are negative. It just didn't happen. And that's, that's the magical thing about working with this group. And what makes me look forward to what we're going to do next.
Matt Coles:Now, having said that, I do want to call out for, for listeners slash viewers, we did have, it was a, there were, it was a well moderated conversation,
Izar Tarandach:Oh, yeah.
Matt Coles:right? And so, thank you, Chris, for providing that moderation and that, that sanity, those sanity checks, to many, in many cases, providing a sanity case, or, okay, we're at, you know, A versus B, can we come to a conclusion? You know, keeping us on track. Uh, you know, even though it didn't come to a vote in most cases, the fact that, that we had control over the process, that there was control in the process so that So that, you know, there were valves to release the pressure, uh, in, in the conversation, not again, not that nobody blew up or anything like that, but, um, you know, it wasn't, it wasn't a free, free for all, and we're all professionals doing our professional job, uh, you know, things that we do that we'd like to do, the, you know, working, working with threat modeling and doing that. Oh,
Izar Tarandach:And in that same note I want to call out our friend Jonathan Marcel, who did an amazing job, not only on the site, on the presentation, but behind the scenes on pushing things around and getting people to come down to a decision on wording and stuff like that. And more than anything, being the maturity policy, police. Many, many times in these discussions we will be talking about something and all of a sudden we would figure out that by almost by Default we fall into a discussion of the maturity of the thing and he would put it back to no, wait, wait, wait, wait You're talking maturity. What's the capability behind
Chris Romeo:You know, the funny thing about that though, is he only represented the maturity police for about two meetings. But then he built this, he built the maturity, the virtual maturity police force, where we all started going back to it automatically. And it wasn't even like he had to raise his hand to kind of get our attention and go, Oh, maturity. We would all be like, Oh, wait, that's maturity. So, so kudos to him for implanting this, this capability. Ah, you like that? Implanting a capability amongst the team where we would self correct when we started to go down the maturity approach.
Izar Tarandach:so notice he turned on the capability but over the calls we improved on the maturity of it By getting better as a team on doing it.
Chris Romeo:So we're level four now when it comes to
Izar Tarandach:we're level 4.
Chris Romeo:let's, uh, yeah, let's bring this document up. I mean, we have the power of the internet, I guess, and we can just take a quick peek at it so those that are watching this on YouTube
Izar Tarandach:Matt is going to read it, to read the screen for those who are listening in the car. How
Matt Coles:the screen. They have screen readers for that.
Chris Romeo:Exactly. You can,
Izar Tarandach:does the screen reader go? Browser, threat modeling capabilities, URL.
Chris Romeo:it's definitely your voice that people are listening to as well. So, uh, just to highlight, I'll, I'll put, I'll highlight some things here and you guys can feel free to comment on them as you think, um, or as you feel led. But for those that are listening from an audio perspective, when you get back to a safe place, please don't try to view the capabilities in your car while driving. Uh, but the document is at threatmodelingmanifesto.org/capabilities. And will find that link in the show notes as well. So if I just thought I'd throw it out there, uh, once again, don't look at it. If you're flying a plane, helicopter, or driving a car or a boat at this particular time.
Izar Tarandach:Subamrines are fine.
Chris Romeo:submarines are fine. Cause you can just put it on autopilot
Izar Tarandach:No, and you don't see where you're going anyway, so
Chris Romeo:That's true. So pointing out the documents. So the first section. We kind of, we have a little bit of introduction and then we're defining what's a threat modeling capability. So Matt did a great job of, of summarizing what exists in this section already early when we first started talking about capabilities. There's a little more context there that, um, that you can, you can get into. And then Izar referenced the seven different categories or the groupings of these process areas of, of. Of capabilities that are strategy, education, creating threat models, acting on threat models, communications, measurement, and program management. And then for each of these individual capabilities, there is a name, there's a little bit of, uh, of introductory text that we like to refer to as the North Star. was what we were guiding towards. What did we want this particular process area to help us get to? And then with each of these capabilities, it's really just a two or three word, uh, descriptive title of what the capability is, and then one to two to maybe maximum three sentences that help to describe what it is. And so I'm going to pick one just at random that we can just have a little chat about here.
Izar Tarandach:Let's get one of the ones that we spent a lot of time on.
Chris Romeo:Okay, well give me, give me a hint as far as
Izar Tarandach:Ah What was that last one? Go down a bit Uh No It was on communications, I think, uh
Matt Coles:one, one thing to highlight is a lot of the time when we were, when we were building these, these capabilities, they move. So the process areas didn't originally start this way. And capabilities moved around and got merged and got split and got remerged. So, you know, if there's one in particular, you know, many, it's hard to call out one in particular that had a lot of conversation because that may have been a conversation around a mature, a capability of existed. Before a split or merge.
Izar Tarandach:No, wait, we had the one that we went on, uh, how the program Was it How the Problem Feeds on Itself, or Remember that one?
Matt Coles:Uh, yeah, the, uh, I, yeah, wow. You know what? This is, it should be fresh in our mind. Um, so,
Izar Tarandach:we, we are just, uh, pushing the thing down because we don't want to remember the discussion. Uh, what was it? Not Continuous Changes.
Matt Coles:well, so while Izar is figuring that out, actually, Chris, why don't we go back to the top? Let's talk through each of the process areas. We can, if there's nothing to talk about specifically, we can skip it. You know, strategy first is really about, you know, Um, overall, what would you want to look for, you know, how would you approach if you don't have a threat modeling practice already in your organization, but also if you have one, how would you potentially look to improve it, make it more effective, right? Um, so we're talking about things like making sure that you have management buy in and that, uh, execution at an organizational level, uh, can, can occur. Right? When we look at, then, then from that feeds a lot of these other, other process areas, right? So, education, which is the next one in our process area list, is all about ensuring that people have the right resources, knowledge, and other resources, so that they are able to be effective in the role and, and that the program can be successful.
Chris Romeo:let's, let's stop here for a second. Let's unpack a few of these, because I think this is, this is just a, a one that's, there's a six pack of, of capabilities here. Um, but you've got, you know, training assignments. So threat modeling training is part of the organization's curriculum. Stakeholders assign resources that everyone in the organization can learn. That is a capability, because we, we want people to learn more about threat modeling, so they can get better at threat modeling. And that goes right along with active practice, where practitioners use experiential, experiential, learning to develop threat modeling skills by performing hands on threat modeling, right? It's
Matt Coles:not, not, not exponential.
Chris Romeo:Experiential, I said it, uh, I had to correct myself there. Um,
Izar Tarandach:away of my logarithms!
Chris Romeo:that's right, I don't even know what a logarithm is, so. But then, and then all the way down to continuing education, a capability that describes how people need to continuously be learning more about threat modeling and security and privacy. And so, that's a package of a process area with education, describing. You know, these best practices and like, like Izar and Matt have already described, like this is a zero or one and Izar was the one who helped drive us to these have to be provable. It can't just be like, we feel like we're doing it. Each of these has something, a deliverable that you'd be able to look at and determine whether in fact a company was, your company was a, was completing these various capabilities.
Matt Coles:it's important to call out that these, this, that training or education process area has a root in the manifesto, right? So journey of understanding over security and privacy snapshot, and how do you make that successful, right? So we want to, we want to ensure. Or we hope that organizations build out a program that is consistent in its values and principles and that education is a key part to that. And so you'll see that as a theme throughout that a lot of the process areas and the capabilities themselves tie back to one of the concepts in the manifesto.
Chris Romeo:Okay. How about creating?
Izar Tarandach:Wait, wait, wait, wait, wait. Go back up. I remember now, the one that we spent the most time on ended up being called conventional alignment.
Chris Romeo:Ah, yes.
Izar Tarandach:Remember? My goodness, we spent so much time on this one.
Matt Coles:Well, okay, so let's tell the full story here. Part of the reason why there was, there was contention on the convention is that,
Chris Romeo:ha, ha. Contention on the convention of the
Izar Tarandach:Not when I'm drinking!
Chris Romeo:you please work another chunk, chunk, chunk type of word into it? The inflection of the convection of the convention was about, more about the
Izar Tarandach:Deflection.
Chris Romeo:deflection of
Matt Coles:more, was more about getting a good, a good two word adjective noun or adjective adverb action. Uh, than anything else. Uh, so, so, so each of the capabilities you'll notice have a format of a word and a word. There's actually a hyphenated word in here somewhere, so it's three, but, uh, it's, it's,
Izar Tarandach:we cheated.
Matt Coles:it's a descriptive action, right? So, in this case, it's alignment of conventions and, uh, and that training reinforces, inform, is informed by, and reinforces, the organizational norms, so that you get consistency where you want it, uh, and that, that consistency improves the capability of the threat modeling program.
Izar Tarandach:I'm sorry, but when we managed to put the sentence infused locality of organizational culture into threat modeling, that's when I felt that we got there. That's when I felt that we achieved. Maximum Dilbertness.
Chris Romeo:Ha, ha,
Matt Coles:And if you're, if it requires an LLM to interpret it and, and understand it correctly.
Izar Tarandach:that's because we used one to write it,
Chris Romeo:We did not. We used a bunch of brains. brains. All right, so then creating threat models is the next section where there's 12 capabilities that are providing, I would say, best practices of things that you should try to do in your process of creating threat modeling, like fostering participation. We want to make sure that we have diversity of job function for people that are, that are, doing performing threat modeling.
Izar Tarandach:Yes.
Chris Romeo:We don't want to just have developers sit around in a corner because when you bring a tester and a product manager and an architect and they all sit at the table, there's a lot of good. And once again, that has a direct point back to the manifesto too. Like a, that's a, there's a direct correlation there right
Matt Coles:and not just the manifesto, if you've seen any of our previous podcast episodes, you ever seen any of the presentations that any of us have done at conferences or through trainings or through, you know, books or other material that we've put together, this is a common theme, right? Having, having threat modeling be part of the process that
Izar Tarandach:it's a team sport,
Matt Coles:Yeah, that everyone is involved with, so.
Chris Romeo:That's
Izar Tarandach:and I,
Chris Romeo:idea. Threat modeling is a team sport.
Izar Tarandach:and I think that it's fair to say that we started with the manifesto, the manifesto sort of drew a line, And people are like, okay, these are important things to do, but, uh, how do I actually do them? What do I need to do in order to do them? And the capabilities is here exactly to show these are the pieces that you need. These are the things that you need to work on having, so that you can achieve the targets that we put out in the manifesto. And the next thing in line probably is going to tell people how to get better at these capabilities.
Chris Romeo:Ah, spoiler alert. Come on.
Izar Tarandach:nudge,
Chris Romeo:finished
Izar Tarandach:nudge nudge, say no more, say no more.
Chris Romeo:All right. So then we've got acting on threat models, um, you know, things that we could do to ensure that the findings of the threat models, people are taking action upon them. So definition of done, seamless alignment, baseline improvements, and then risk management. Then you've got communication. What are some best practices for communication and measurement, and then ultimately program management as. You know, ways to, to, you know, to, to improve the threat modeling program, which is driving all the other good stuff that's happening in, uh, in our, with our threat modeling. It is our ultimate goal. So there's the list of our, of the 15 people. So we had a few people from the manifesto drop out due to other commitments. And then we had a couple of new folks that joined us, like Sarah-Jane Madden, who Sarah Jane had a, just a crucial, um, Just a different viewpoint. And, um, I'm the one that, that added, that tried, that invited her and told the rest of the team, cause we sat in Dublin last year at the OWASP conference and sat and listened to her talk. I think he's our, you and I were actually sitting there, right? And we're, we were like, she thinks about this differently than anybody else I know, which is great.
Izar Tarandach:She's one of us.
Chris Romeo:She's brilliant!
Izar Tarandach:She's one of us.
Chris Romeo:and she thinks about it. She just thinks about it differently. uh, that's what we wanted. We wanted, we don't want this group to be represented as we all think the same way and conclude the same way we want different opinions here. And so, uh, Sarah Jane was a new, a new person that we were able to bring into the mix, um, Seba. who is very well known in the OWASP world for his work on SAM
Izar Tarandach:Mm hmm.
Chris Romeo:and lots of other things. But, uh, we were able to, to invite him to join us. He's also passionate about threat modeling. We were able to invite him to join us and bring his experience with SAM into the mix as we were trying to figure out maturity models versus capabilities and stuff like that. So it was great to have some new folks join, uh, join us in the mix. Now let's talk about as our final point of discussions slash argument. Um, cause now we're not in the confines of the threat
Izar Tarandach:wait, wait, wait, before we go there. We have to mention Sheila, that she's already, uh, uh, she's already, uh, uh, honorary member of the group. And, uh, by now, after reading so much about threat modeling, guys, if you need somebody to give you training, I think that Sheila, by now, can do it.
Chris Romeo:Yep.
Izar Tarandach:So, yeah, no, without her editing powers, uh, we would be stuck words meeting for a long, long time, and it wouldn't be nearly as good as we think it came out. So,
Matt Coles:have been reliant on the hallucinations of some LLM somewhere.
Izar Tarandach:that. Yeah. And, uh, grammarly. Pfft.
Chris Romeo:Yep. So yeah, thank you. Definitely. Thank you to Sheila for, uh, for, for being our editor and turning the document around fast.
Izar Tarandach:and our guy Lio ti
Chris Romeo:And you have to figure that one out yourself, folks. So ask,
Izar Tarandach:of the Of the Legions.
Chris Romeo:chat GPT, maybe it can help you with this. So, all right. So we started this conversation when we started this group working again, discussing and debating. Capabilities versus maturity model and we had, we had some good debate and conversation early on the process and eventually we landed on capabilities. So where, so, so what, what would a maturity model as the next chapter of this book look like since you were a person that was very on the forefront of capabilities and maturity models?
Matt Coles:well, so if anyone is familiar with what is, you know, the capability maturity model, Uh, system, right? So CMMI, uh, is, is available. It's in two dot something now. Um, so you'll, you'll, you may, you may be familiar with that if you're familiar with CMMI, uh, or if you're familiar with anything out of the open group, um, especially, um, uh, they have, have capability maturity. Capability model, uh, so capabilities are things that you do or do not do, uh, whereas maturity is the, is the how well do you do those things, right? So, so if you look at a capability, um, you know, if we pick any of the capabilities, really just, uh, you know, there's, uh, let me just, I'm going to just go back and pick one of these, right? So if we look at, um. At life cycle integration, just because it's strategy, right? Um, a little, a low, a low maturity of life cycle integration would be very ad hoc. You know, maybe the engineer or security champion or somebody working in a, in a, on a product within a life cycle. May, may just, Oh, I want to do threat modeling and they add it to their, to their task list, right? That's, that's pretty low maturity. Um, but at a higher level of maturity, uh, you're going to have it baked into your lifecycle definition. And so it becomes a regular activity. Um, and then at an even higher maturity to that, uh, there may be some sort of feedback loop. And so it may be become an adaptive, uh, uh, uh, Adaptive activity within the life cycle. So the, so the, the way that the fact that you do some sort of integration gives you the capability and then how effective you are, how repeatable it is, how measurable it is, how adaptive it is, is what you gain through increasing levels of maturity.
Chris Romeo:Yeah, I think that's, that's the next logical step
Izar Tarandach:Yep.
Chris Romeo:is to go to, to take the capabilities and. Consider what maturity levels would look like for each of them and
Matt Coles:so what, oh, sorry, so just
Izar Tarandach:No, no, go, go.
Matt Coles:One of the big cut... parts of the conversation we of course wanted to, we had and want to keep in mind when we look at. You know, how would we put a maturity model together? You know, if we, if we take that as the next step, is there are a number of maturity models today that exist that are, um, that have varying degrees of, of, of success and, and value, right? So you have. OWASP has a maturity model in SAM, right? So the security assurance, uh, maturity model where for each capability area, they, they offer a, you know, step one, step two, step three, right? So maturity is fixed with fixed definitions. You have BSIM, you know, build security and maturity model. Well, it's not maturity model. It's something else, uh, that I think it stands for, um, that. describes sort of a set of things that people do as a measure of maturity, but not necessarily defining what those things should be. And then you have CMMI, which has for each process area or capability, Um, an understanding of a 0 to 5 scale where increasing maturity means something and, and, and may not actually define what it means to be at level 1 versus level 3 versus level 5, but has a, has a goal in mind. And so, um, what, what is, valuable for us as threat modelers and as organizations that are implementing a threat modeling program. Uh, you know, is it appropriate to say, well, at level 4 or level 5 of life cycle integration, you should have an adaptive threat modeling process without actually telling them what that means versus making it prescriptive and potentially missing the mark, right? So, those are some of the things we will, that will have to be worked out if anyone's going to build on top of these capabilities.
Izar Tarandach:Yeah. But one thing I was thinking about the, the. The maturity side of things is that, as you said, for example, the BSIM, it looks at different verticals. But at the end of the day, it's the same things that they're looking at and measuring. They just check who has more of them in which vertical. And for us in here, I think I'm just throwing, like, thoughts out, not, not, not talking for the group. This is probably going to be an interesting discussion. Does it mean that we always have to have the, the, the same number of levels for each one of the capabilities? Does it always mean that everybody has to migrate towards the higher levels? Or if you are in a specific kind of organization, you can stop in the middle and everything's going to be fine. So.
Chris Romeo:You can do that with any maturity model though, like that's a personal, personal choice
Izar Tarandach:but that's a personal choice.
Chris Romeo:You're not forced to go to a highest, the highest level. I mean, the highest level should define what we as a group of practitioners think is the best state for that particular capability based on our collective experience.
Izar Tarandach:the thing is, going back to our recurring theme of, uh, being reasonable, if we go with a bad, good, best, do we always have to be at best? Is the reasonable level being always at best? Probably not.
Matt Coles:you're starting with a, you're starting with a fundamental flaw in your, in your reasoning. I
Izar Tarandach:is,
Matt Coles:bad, good, best.
Izar Tarandach:yeah.
Matt Coles:I don't think we, I don't think the maturity is a, is that type of measurements.
Izar Tarandach:Oh, no, no, I'm just throwing it. I mean, bad is not having, good is having something, best is having the whole thing. But, uh, just throwing something to have something to measure here. It could be level zero to level five. So,
Matt Coles:it's not that it's your, you're, you're associating value to those levels.
Izar Tarandach:interesting, interesting. So you're already saying that being at a higher level doesn't mean that you are at a higher value of the thing.
Matt Coles:It may not be
Izar Tarandach:just means that you are better at doing it.
Matt Coles:More consistent, more consistent. You may have higher assurance,
Izar Tarandach:Nice, nice.
Matt Coles:necessarily even mean that you're doing, you, you could have, so similar to security, right? You can have high assurance and still have a Swiss cheese product,
Izar Tarandach:so we have, we have to, we have to definitely be quantitative rather than qualitative in the measurement of that maturity. Okay, somebody write that down.
Matt Coles:But it's what you do. It's what you do and how you do it, not necessarily that you, you know, you could, you could still find a billion security threats through an adaptive process. The adaptive process is the important part, not necessarily the, not necessarily the outcomes. The outcomes should be manageable. The outcomes should be understandable. The outcomes should drive behavior, but not from the purpose of making, saying maturity three versus maturity five is better or worse.
Izar Tarandach:Are you telling me process over outcome?
Matt Coles:So, this is kind of a fundamental flaw of maturity models that why people tend to probably don't like it so much. But yeah, a little bit.
Chris Romeo:Everything needs to be about outcomes though.
Izar Tarandach:No, no, no.
Matt Coles:Well,
Izar Tarandach:Actually, we should make an episode on that. It's definitely process.
Chris Romeo:from a business perspective, it's, it's about outcome. At the end of the day, if I don't have outcome, I don't have a business. And so you can make all the process you want, but there's no, you're going to go to an empty building because we don't exist anymore.
Izar Tarandach:You're going to have an outcome either way. Having a process to get to some outcome means that if that outcome is not the optimal one, you have something to go back and debug to get to the optimal one. So the process is actually more important than the outcome.
Matt Coles:And you'll have
Chris Romeo:I'll argue on the case of chaos.
Matt Coles:have predictability. That's actually, that is literally the choice. Do you want chaos or do you want predictability? Um, so if anything, a process maturity model is all about capability, the process maturity model is entirely about repeatability and predictability,
Izar Tarandach:Mm hmm. Ah,
Matt Coles:but how you're getting
Chris Romeo:So you can refer to me as Tyler Durden from now on. A
Matt Coles:And by the way, if, if anyone is listening to this, if anyone, any of our viewers are listening to this and have thoughts on capability and maturity models, absolutely we'd love to hear from you.
Izar Tarandach:yes, please. Especially after you read the whole thing.
Chris Romeo:Yeah, that's a good way to segue into the, the end of this episode. So, uh, we definitely want to get feedback on this thing. Like we didn't, we didn't create this thing on an, in an ivory tower and think that we somehow have all the answers. Uh, we want people to analyze it and then provide comments and feedback. Uh, you can do that. Uh, there's a GitHub repo for the threat modeling manifesto and now the capabilities. So you can, uh, give us, send us issues from there. Um, you can ping any of us directly, we'll, we'll pass the feedback on to the rest of the team. So feel free to, uh, to give us that feedback. We really do want to know. What we got right, what we, what you think we might've gotten wrong so that, you know, we can revise it in the future and, and make an even better thing.
Izar Tarandach:What we forgot.
Chris Romeo:What we forgot.
Izar Tarandach:Very
Chris Romeo:a good thing too. Like we, yeah, we, I wouldn't, I would not guarantee that we captured everything that needs to go into the essence of a threat modeling program.
Matt Coles:Yeah. Or, or, or
Chris Romeo:we missed.
Matt Coles:Or what ended up on the cutting room floor.
Chris Romeo:True.
Izar Tarandach:that one too.
Chris Romeo:That's
Matt Coles:What did we, what did we, what did we deprioritize because of our collective understanding, and then, but we should have kept?
Chris Romeo:Yeah,
Izar Tarandach:yes, this is important. What an interesting question. Now?
Chris Romeo:to leave that, we'll leave that for our audience to ponder. Uh, hopefully you enjoyed walking through this, uh, threat modeling capabilities document. I know we all had a blast putting it together, but I am glad that we shipped it finally. Uh, that was a momentous occasion to ship it, get it out to the world so we can gather feedback
Izar Tarandach:Now? Now?
Chris Romeo:Yeah. Let the record show, Izar is celebrating with fireworks in his camera display for anyone driving the boat, the plane. the helicopter or the car at this point who's not able to look at to the video screen. So once again, thanks for joining us on The Security Table. We will be back next week with some other wacky zany fun.
Izar Tarandach:We have the capability.
