How I Learned to Stop Worrying and Love the AI

Dive into the contentious world of AI in software development, where artificial intelligence reshapes coding and application security. We spotlight the surge of AI-generated code and the incorporation of copy-pasted snippets from popular forums, focusing on their impact on code quality, security, and maintainability. The conversation critically examines the diminishing role of traditional quality assurance measures versus the growing reliance on automated tools and AI, highlighting potential compromises between development speed and security integrity.

The discussion broadens to consider the future of software security tools in an AI-dominated era, questioning whether AI-generated code could make static application security testing (SAST) tools obsolete or introduce new challenges requiring more human oversight. The debate intensifies around the trustworthiness of AI in handling complex business logic and security policies without introducing vulnerabilities.

The dialogue concludes by reflecting on the balance between innovation and caution in software development. As AI advances, the conversation centers on ensuring it enhances rather than compromises application security, offering insights, anecdotes, and a dose of humor along the way. Stay tuned for more thought-provoking discussions on the intersection of AI and software security.

Helpful Links:
Article: "New study on coding behavior raises questions about impact of AI on software development" at GeekWire -- https://www.geekwire.com/2024/new-study-on-coding-behavior-raises-questions-about-impact-of-ai-on-software-development/

FOLLOW OUR SOCIAL MEDIA:

➜Twitter: @SecTablePodcast
➜LinkedIn: The Security Table Podcast
➜YouTube: The Security Table YouTube Channel

Thanks for Listening!

Transcript

Matt Coles: 0:09

Ooh, Oh, this is cool.

Izar Tarandach: 0:14

What? What? What did I miss?

Chris Romeo: 0:15

it to the world?

Izar Tarandach: 0:17

What did I miss?

Chris Romeo: 0:18

Go ahead, we're rolling. So let's see, what is the t shirt? Watching the Watchers. Oh, that's,

Izar Tarandach: 0:27

Oh,

Chris Romeo: 0:27

What was that show, Fringe? It's making me think of the Watchers, but

Matt Coles: 0:32

there, it's on the screen.

Chris Romeo: 0:33

Yeah,

Izar Tarandach: 0:36

yeah,

Matt Coles: 0:37

It's a very, uh, it's, it's so not like the EFF.

Izar Tarandach: 0:41

yeah,

Matt Coles: 0:43

It's very, uh, Illuminati, uh, kind of

Izar Tarandach: 0:46

I wouldn't wear that to certain places.

Matt Coles: 0:49

I would absolutely wear this to every place.

Izar Tarandach: 0:52

No, there are other where you wear that. People say, hey, where did you get that? And you say, oh, I just got it at the last meeting.

Matt Coles: 0:59

Like I'm working, I'm wearing that work to work. If I ever have to go into the office, I'll wear that to work.

Chris Romeo: 1:05

That's right. Matt is, Matt is all about freedom. He's a big freedom guy. All right, well,

Matt Coles: 1:12

electronic freedom.

Chris Romeo: 1:13

Electronic Freedom.

Izar Tarandach: 1:15

Freedom!

Chris Romeo: 1:16

I think, I think you're generally a freedom proponent

Matt Coles: 1:19

Yeah, generally, generally.

Chris Romeo: 1:20

other than your dogs, which are not allowed to be free and

Matt Coles: 1:23

No, that would

Chris Romeo: 1:24

roam the

Matt Coles: 1:25

wouldn't end well. Because then, because then, I wouldn't be free. What

Chris Romeo: 1:35

dogs would be free. All right. Well, on that note, welcome to another episode of The Security Table, where you kind of have joined a conversation in progress, but Sometimes that's okay, because you're getting a behind the scenes view of what happens in the lives of myself, Chris Romeo, Izar Tarandach, and Matt Coles. So, welcome again to, uh, Security Table, where we're gonna focus in on artificial intelligence. A topic that is on many people's minds these days. And no, it's not robots. Izar, Wondered sometimes if is a robot in certain situations. I was like, is this guy a robot? I don't think he's a Terminator because I haven't really seen him like running and turning into metal or running fast or anything. But

Izar Tarandach: 2:17

It's worse. Sometimes I wish I were

Chris Romeo: 2:20

I mean, we all sometimes wish we were a robot, but that's, that's a story or an episode for a future

Matt Coles: 2:27

is it? Domo arigato, Mr. Roboto. I can't, I, we can't use the whole clip, I don't think, that would demonetize, but uh,

Chris Romeo: 2:36

yeah, that was a good, uh, that's a good song though. I it's that's seventies, right?

Matt Coles: 2:40

70s, late 70s, early 80s. Though, well, late 70s, early 80s, but yeah.

Izar Tarandach: 2:45

No, no, no, it's not late 70s. Don't, don't make me that old, please.

Matt Coles: 2:48

I have to look. While, while we're talking, I go

Chris Romeo: 2:51

I got news for you,

Izar Tarandach: 2:52

no, I'm already looking. 1983. Yes,

Chris Romeo: 2:59

well, I was just a tiny baby at that point. Well, maybe not, but There we go, this is called an ear wig, which will now be stuck in people that are in Australia and also in the gym at this moment in time.

Izar Tarandach: 3:14

it's a great gym song.

Matt Coles: 3:17

Mm hmm.

Chris Romeo: 3:18

don't know, that really help you to push more weight up in the air?

Izar Tarandach: 3:22

Yeah, because gets so peeled that's in your head.

Chris Romeo: 3:27

You're so mad. You're like, put more weight on

Izar Tarandach: 3:30

just like,

Chris Romeo: 3:32

me go for a little bit more weight. is a song's driving me batty.

Izar Tarandach: 3:37

me more weight.

Chris Romeo: 3:41

People are listening to this going, is this an actual show or are these guys just their

Matt Coles: 3:48

Did they hit record too early and they don't know that they're on air?

Chris Romeo: 3:51

Yeah,

Izar Tarandach: 3:52

it's Friday. We're allowed to lose our minds.

Chris Romeo: 3:54

that's true. That's true. Well, let's get out. Let's get ourselves on, uh, on, I can't even think of the word track.

Matt Coles: 4:00

gonna lose our minds completely here.

Chris Romeo: 4:02

Yeah, I need to, you know, I need, I need to get my inner, uh, my inner actor. I have to tap into that which what's, what's my motivation and my role. Oh, angry security person. Got it.

Izar Tarandach: 4:13

Done that!

Chris Romeo: 4:14

yeah, my normal daily routine. But so we want to talk about AI. and we specifically want to talk about this study that was, that's recently come out. Um, we'll put a link to it in the show notes, uh, the actual description of the study. The title of it is new study on coding behavior raises questions about impact of AI on software development. And, uh, a couple of notes of things that I kind of took away from this article just to set the stage. Uh, they looked at 153 million change lines of code. Comparing changes done in 2023 versus previous years, uh, before AI hit the scene here. Um, they, they found that code churn or percentage of lines thrown out less than two weeks after being authored is on the rise. This is expected to double in 2024. So people are putting code in and then it's getting ripped out. Percentage of copy pasted code is increasing faster than updated, deleted, or moved code. Their conclusion is this is in response to AI generated code, which is then being pasted in. And then really they kind of summarized it as AI coding assistants are very good at adding code, but they can cause AI induced tech debt. So there's a whole other new, uh, new kind of a term there. So I guess,

Matt Coles: 5:35

Yeah, and I wonder, so just to jump right into the deep end on that one, is it, is it technical debt, like we would traditionally consider technical debt of bugs, defects, you know, things that have gone wrong, that would require either redesign or rework later, um, you know, refactoring later. But, but I think what we also have been starting to, at least I've been starting to hear about is, um, uh, things like copyright. copyright infringed code, right? So it isn't just technical debt, it's potentially liability debt

Chris Romeo: 6:15

why would that be? Because it's coming from an AI that's been trained on something copyright and then it's passing through and getting pasted. Is that, is that the

Matt Coles: 6:23

Yeah. Yeah. So, I mean, I, I can imagine, I, I think if I remember correctly, you know, GitHub, you know, public, public GitHub sources and, and, or public repositories for source code may have been part of training sets. Some of that might be under GPL v3.

Chris Romeo: 6:40

Hmm.

Matt Coles: 6:40

And now you take that code snippet and you drop it into somebody else's code base. And now that, that comes along for the ride. Right. And this is the stack overflow problem, right? Also the developers even before AI, right? Developers would go to the stack overflow or other popular coding sites and say, Hey, how do I do this? Oh, here's the code snippet. Nevermind there's a license attached to that code.

Chris Romeo: 7:00

Yeah. And, uh, and also the fact that that code was likely horribly, horribly insecure, which is the, what we think of as the Stack Overflow problem. But then you got to wonder who is training their AIs from Stack Overflow, because find me a bigger, more prevalent, database of coding problems and solutions on earth. I can't think of one that's bigger than Stack Overflow's database of, of all of the issues that they've tracked over the years. But if you're taking that and feeding it into a model and using it as training data, we know from various, there's been academic studies that have shown this, that, uh, they've profiled lots of different issues there. And often the least, one of the least insecure options is the one that gets, Upvoted the most, and that there there'll be a secure option down at the bottom that's gotten three up votes. And the insecure easy one has gotten 10,000 up votes. And so if that's your training, debt training set, you could potentially, that could potentially be what, what are some of these challenges that this other study's finding?

Izar Tarandach: 8:08

So before I, I put my foot in my mouth, we're talking about code that's AI generated right now, right?

Chris Romeo: 8:16

Yes.

Izar Tarandach: 8:17

Okay, so Stack Overflow, the problem with Stack Overflow is that, as you said, Chris, we get snippets. My problem with AI generated code is that all of a sudden we are getting pages and pages and pages.

Matt Coles: 8:32

Big snippets, big snippets, very big

Izar Tarandach: 8:33

Big snippets, very big snippets, the biggest snippets ever. And the thing is that in the past few years I have experienced a number of times, uh, developers moving from a monolith to a service based architecture. And what happens many times is that pieces of code that people look at them and say I don't understand this because Joe who wrote it is not, has not been with the company for a number of years. And when I cross those two streams, what I guess is, what I get is that now we are building from scratch pieces of code that nobody understands because Joe that wrote it is an AI.

Chris Romeo: 9:16

Hmm

Matt Coles: 9:17

And it's no longer here.

Izar Tarandach: 9:19

And no longer here. And if I ask it to do the same thing again and explain to me, it's going to do a completely different piece of code that probably does the same thing, but won't be able to explain that one.

Chris Romeo: 9:29

So there's a lack of predictability that AI provides, which you're saying is causing a challenge in developing software with ai.

Matt Coles: 9:37

Repeatability. Repeatability, probably more so than predictability, right?

Chris Romeo: 9:42

Okay,

Izar Tarandach: 9:43

it's,

Chris Romeo: 9:43

yeah.

Izar Tarandach: 9:45

I think that it's even worse than that. Uh, remember 10, 20 years ago the thing with pair programming? think that it got too far, but

Matt Coles: 9:57

that was the extreme programming, uh, that was an extreme programming method, right? So

Izar Tarandach: 10:01

right, it was a thing for a while. I remember that I went to to check out a place in San Francisco and they had like this long tables, and no computers on them, and people were supposed to come and two people per laptop and work like that. Which is nice, hey, it's two people talking over code, right, and discussing every single line, and right here, right there, give me the mouse, give me the keyboard, no, I'm going to close this thing and drop it in your head. Very, uh, very interesting, uh, debate, but now we are trying to do the same thing with AI copilots and whatnots. How much of a conversation can you have with a thing that's supposed to have all the knowledge ever created about writing lines of code, ever? And how good are you when you are talking to this thing, even in natural language? And I'll be the first one to say, sometimes I find it very, very difficult to express to other people in natural language what I want them to do. So to a machine, it's probably even harder. I don't know, I'm very It's an interesting position to be because from one side, I want the technology to work because A, it's wow, so cool. B, while there's so much promise in here, both in time, even in security and all that good stuff. But on the other hand, I'm like, we had difficulties dealing with software when people actually understood what they wrote and how it worked. Now we are going to drop a lot of agency onto something that we basically don't understand exactly how it works, but we like the result so we're going to keep using this thing. And

Matt Coles: 11:46

the study highlights actually, I think what, the one, the point is the points you just raised. So the study calls out specifically copy pasted code being treated as. As if it was a short term developer, right? So it's somebody who comes in, you know, we talk about this all the time, the parachute consultant, you know, the consultant who parachutes in, looks at something, goes, Oh yeah, here's your problem, drops a block of code in and leaves,

Izar Tarandach: 12:12

the C go,

Matt Coles: 12:13

And then, and then you have to pick up the pieces because you either have to integrate it and figure out how to make it work, or it it's now part of your code base. And then later when you have to go support it, well, what do you do? You have to have, does the, do the developers who are working on the project know what, what, what they're supporting, uh, at this point. And, and I think in nowadays, especially, they're going to turn to maybe another AI system to say, summarize this for me. Or help me interpret this.

Chris Romeo: 12:49

so we

Izar Tarandach: 12:49

then going to get,

Chris Romeo: 12:51

now gonna say, we could be, we could be facing a world where, based on what both of you are describing here, software is so complicated that only the AI can parse it. Understand what it's doing, and then we get it back into that trust problem that we've talked about. It seems like every time we talk about ai, we end up coming back around to the trust issue, and that's really the trust issue if only the AI can understand the convoluted code. But the, the counter to that is we always had that. We, we've always had that time in our career where somebody writes some code and we're like, you look at it and you're like, I don't even know how this. Compiles,

Izar Tarandach: 13:32

that's true,

Chris Romeo: 13:33

less

Izar Tarandach: 13:33

and that, that's true, and

Chris Romeo: 13:35

does it. And it does the job. Cause I know I wrote a piece of code of this and in security journeys platform many years ago in the beginning, it was an XML parser. It was a disaster. wrote this thing

Matt Coles: 13:45

Well, that was your first, that was your first mistake right

Chris Romeo: 13:47

Yeah, but I wrote it and like, it was, and it's, it just worked. And, and the other developers are like, uh, how does, how does that work? I'm like, ah, it works. It just, I did it

Izar Tarandach: 13:57

my problem. My problem is not even the complexity, it's the corollary of, uh, I don't know how it works, but it works. The next sentence is the one that bothers me. Don't touch it. Because if you touch it, it's probably going to stop working, so don't touch it. We don't know how it works. But, no, no, you go.

Matt Coles: 14:17

Well, I just wanted to jump in with a thought, just a question, sort of a random thought came to head. What's different here? So. We expect developers to make mistakes, right? And we expect developers to have certain behaviors like, Oh, I wrote the piece of code, I don't know how, I don't know exactly what happened there, it compiled, don't touch it, it's fine. We expect those behaviors of developers, of humans. We now have humans who are relying on other developers who are not developers, they're AI, or we're relying on AI to do things as developers. But what's different here? Is it because it's an AI system? Is it because of that lack of trust? Is it that lack of predictability or repeatability? These, these parameters, these things that, um, that we don't understand, uh, or, or have, have questions about. And then I would also tie that to, uh, so when we were talking about this episode and, you know, getting ready for it, I was talking about sort of the, the, the, The introduction of code and then the validation of that code. And so we're, we're sort of at a confluence here where we've, not only do we have developers, we have developers introducing code from other places that are not theirs, uh, you know, AI generated, or we have developers who are not developer, not humans, they are AI, but we've also over the, to get to this point, remove the safety nets. From the development process, right? We've taken out QA, we've taken out, uh, checks and balances in that process, other than code check in to some sort of analysis tool, to a, to a PR, you know, automation.

Chris Romeo: 15:58

But that's not, that's not AI specific, right? We've

Matt Coles: 16:01

That's not AI

Chris Romeo: 16:01

DevOps, DevOps has done to

Matt Coles: 16:03

ex Exactly, right. So we've, we've taken away the safety net and the checks and balances that would allow us to establish trust and assurance. Let's go back to security terms here from whether it's developers or machines introducing code into the code base. And, and, and by the way, we're talking about code here, but it could be in introducing components. Hey, I'm going to pull component X or Y or Z out of the, out of the air and drop it here into this build. Right. Um, or, or even The components that are used by the build system, right? All of these things are probably in scope for this discussion. We remove all the safety nets.

Chris Romeo: 16:37

I want to, I want to just, want to throw a big question on the table here. And Izar, did you want to finish a thought on this particular, what Matt

Izar Tarandach: 16:45

Yeah, no,

Chris Romeo: 16:46

I want I want to go in a different direction, but I think I want, I want to throw a big, a big problem on the table.

Izar Tarandach: 16:51

so wait, I, I just wanted to, to complete Matt's thought with something that we wrote in the, the fish book. that developers move at the speed of innovation, but security moves at the speed of caution. And I think that that's what we are showing here. They want to run forward as quick as possible, get to the shiny thing in the hands of the consumers as soon as possible. And we are pulling back and putting back and putting back. And again, we are not saying don't do it. We are saying do it, but exercise caution.

Matt Coles: 17:25

We're sort of like the, we're the, we're the, we're the restraint, we're the restraint, um, cable on an aircraft carrier. We want the plane to land, but we want it to not fall off the edge in the process.

Izar Tarandach: 17:37

so that's not a good analogy, if the cable fails, you expect to happen is the plane to open the throttles to afterburner and just keep going out of the other side.

Matt Coles: 17:47

I was gonna,

Izar Tarandach: 17:48

don't that that's what we

Matt Coles: 17:49

I was gonna say the, it's the parachutes, it's the parachutes on the space shuttle when it lands, but since the space shuttle hasn't flying recently, that's, that was hard. Uh,

Chris Romeo: 17:57

so

Matt Coles: 17:58

your

Chris Romeo: 17:59

my, I want to throw this issue on the table now. It's going to bring together a few things that we talked about. How do guardrails and paved roads differ in an AI developer world? Or do they differ? Do we, do we institute different guardrails? Matt kind of started me on this thread with his thought and then I just started thinking about, I've been thinking about guardrails and paved roads a lot lately. How are we, how are they different or are they different when we have AI software engineers that are not humans?

Matt Coles: 18:32

So I think there is absolutely an opportunity here for establishing guardrails and paved roads. So one of the problems we're talking about is, uh, developers who, Either willfully ignore or don't understand, you know, their, their, maybe if they work for a company that many of the companies will have policies about where you get, where you get your information from, or what tools you're using, or if you're an open source developer, you just don't have the time or you or whatever, and you don't, you know, teenager in their basement working on some piece of code, they're going to go out to it ChatGPT or some random LLM or whatever, you know, BARD or Gemini or whatever it is these days, put something in, get some code out, be done. Guardrails and paved roads, we have an opportunity to have, say, a very specifically trained model for that same AI system where, uh, maybe that's based on coding conventions of the, of the company involved. Right, so now, instead of getting random code, you're now getting code that at least thought, maybe it's random code, but now fits in a certain pattern. Or maybe it comes with validation checks, right? And so, go ahead Izar,

Izar Tarandach: 19:49

No, no, no, no, no, no, you keep

Matt Coles: 19:50

no, no,

Izar Tarandach: 19:52

no, you're going exactly where I want you to go. Go.

Matt Coles: 19:54

so we established the guardrails by, by, by making, making it possible for developers to still do what they think that they need to do in this case. Don't let the AI just return arbitrary answers. If possible.

Izar Tarandach: 20:12

Yeah, and that's great. That's one interpretation. I'm going to offer one that's slightly different. One of the big problems that we have with guardrails and paved roads today is verification. We ask people do it like this, don't do it like that, but we stumble on the verification many times. Now, Matt touched on, on, on,

Matt Coles: 20:40

Icon too. looked rumbled

Chris Romeo: 20:43

Uh, we're gonna have to hang up this call right now because I think the AI has perhaps taken over the world and I'm slightly

Izar Tarandach: 20:49

is it Skynet Day? So the,

Matt Coles: 20:53

This this is, this is just mundane Bluetooth problems in a smart speaker that didn't how, didn't know that was turned off at the

Chris Romeo: 21:00

all right, Skynet, Skynet, stand down, Skynet, we can continue the episode, Skynet is not in effect, okay.

Izar Tarandach: 21:06

so the, the, the place that I was going with this was that A while ago we talked about pair programming and the difficulty explaining yourself to another person and now explaining yourself to an LLM. So basically, using the right prompt, right? Now, Matt touched on the guardrails existing as part, as part of the model, if I understood you right, or part of the framework that's actually creating the, the code. And here I'm going to, uh, exhibit my proud ignorance of, uh, internals of LLMs, and please, if we happen to have any kind of Listener out there that knows this, please, please, please feel free to correct me. I will welcome that very much. The things that I don't understand, I don't think that the guardrails actually get built at that stage. The guardrails would be built either in form of a prompt that we would give before the prompt, like an internal prompt that happens before the prompt that you actually ask the problem. So it's like this preamble to the prompt that becomes part of the prompt. So for every request of, uh, writing code, make sure that you don't introduce a buffer before. Okay. So that thing exists

Matt Coles: 22:21

Yeah. Pre, pre, pre promptt. Pre promptt. I,

Izar Tarandach: 22:23

Then comes your prompt.

Chris Romeo: 22:24

You're putting, but you're putting the trust, you're putting the security control inside of the brain of the

Izar Tarandach: 22:35

Yes. That's where I'm coming. That's where I'm coming. where I'm

Chris Romeo: 22:38

we would never do that in, we would never give the developer in the real world the ability to say, uh, you just write on a piece of paper whether you think you did a good job and it was secure. or no, we we, we would, we measure it ourselves.

Izar Tarandach: 22:51

but, but look at it now measure. Okay. Keep this one on the side for a second.

Matt Coles: 22:55

oh, hold, hold. Hold time, time off for just a sec. Chris, have you never heard of the term attestation? Self-attestation?

Izar Tarandach: 23:09

open!

Chris Romeo: 23:10

Yes.

Matt Coles: 23:15

That was me pulling a pin out and throwing a hand grenade, or pulling pin and throwing the pin, pulling the pin and then leaving the hand grenade. I'm not sure which.

Izar Tarandach: 23:23

wow, wow. Let's talk legislation. No, let's not. Anyway, so the way that I understand the thing, and a while ago there was a bit brouhaha about people being able to extract from ChatGPT, I think, the whole pre prompt, the whole preamble to the prompts, and end. There's a lot of stuff in there, but for example, a lot of, uh, I'm going to put it in quotes because I don't want to go in there. The censorship that's put on top of the results that come from ChatGPT exists in that prompt, right? So to the issue of controls, there are already some controls that live in that pre prompt. Because people are not supposed to have access to that, are not supposed to have ways of changing that. Prompt injection aside, but that's not a security problem as I say. My point is exactly what what Chris said of the measurement. If things keep going the way that they are going At some point somebody's going to wake up and say, why do I need a SAST? The AI is going to create code that's already SAST-ticized. Why do I need, uh, uh, uh, uh, uh, uh

Chris Romeo: 24:29

a term? Sasticized?

Izar Tarandach: 24:32

when do sanitization, that's when you do sanitization with SAST. You saw it first here, I'll own that

Chris Romeo: 24:38

Love it.

Matt Coles: 24:39

that, that sounds like coat cancer. Well, well,

Chris Romeo: 24:44

I

Izar Tarandach: 24:46

You wouldn't be too wrong. But anyway, or people are going to say, Hey, you know what? We don't need SCA anymore because the AI is going to only suggest libraries and packages that are actually good. So these industries are going to go up. These two industries are going to go down and is happen somewhere else. And what

Chris Romeo: 25:04

that's That's such a terrible scenario though, because you're putting all of your trust into something that you can't even explain. Like, guardrails exist because developers are gonna, we want to give them the freedom to do cool stuff and innovative stuff, but they gotta stay within the boundaries of these two markers. You can't go off the side of the mountain with your innovation. How, that,

Izar Tarandach: 25:27

are

Chris Romeo: 25:27

described is not, doesn't, there's no control there. It's all, you're putting all of your trust into the model, into the

Izar Tarandach: 25:33

And you are, no, no, wait, wait, wait. totally right. But have we ever seen people do something like that? Hmm. Hmm.

Matt Coles: 25:42

so I'll remind you, remind you both that we had somebody on who talked about, uh, this exact topic when it came to things like, uh, AI being used for military applications, as an

Izar Tarandach: 25:54

Yeah, true,

Matt Coles: 25:55

So we had a Hack-A-Prompt, Hack-A-Prompt guy, right, who was on, um, great episode. If people haven't listened to they should definitely go listen to it. And, uh, Uh, just a point of correction, I think, Izar, what he highlighted was there was both pre injection, pre prompt filtering and post output filtering, so they do both, uh, and, and in, in the, in that chain, there's a chain, potentially a chain of Uh, input filtering so that you don't get prompt injection, um, and then you get the, the actual prompt execution through the, through the model. And then you have output filtering and potentially validation through another AI system before it comes back to the user. so yeah, go ahead. If you're,

Izar Tarandach: 26:39

what I'm hearing from you is that in between the prompt and the actual generation of the code that's coming back to me, and then in between that generation and the code that's coming back me, there's a number of different, uh, uh, filters that

Matt Coles: 26:51

there

Izar Tarandach: 26:51

no control.

Matt Coles: 26:52

there might be. Well, then maybe that's the guardrail though, right? So the, the, and the Pandora, the paved road. So,

Izar Tarandach: 26:58

good and a

Chris Romeo: 26:58

can't, it can't exist in the same. I can't, I can't accept the fact that the guardrail lives in the same system that the decision and the things are being generated from.

Matt Coles: 27:09

well, so why does it have to? So we're talking

Chris Romeo: 27:12

saying it can't, I'm saying it I can't

Matt Coles: 27:14

It shouldn't, it shouldn't,

Chris Romeo: 27:15

can't trust something that's a box that I just, I put a request in for code and I get something out. And then it has a checkmark next to it that says, Don't worry, it's secure.

Matt Coles: 27:26

Would you feel, would you feel better it, if your pipeline, if your pipeline was, you, you have an IP address that you, if you have IP address that you ask a question to, and then you get, have a different IP address that you get, get the response on, would that make you feel better? I mean,

Izar Tarandach: 27:42

no, but, but, wait, wait, wait. But, but, Chris, see, because of the way that we want the guardrails to be incorporated into the design and the implementation of what we're asking for, the guardrails have to be part of that, for lack of a better word. word, taught, that drives the code that's coming back to you. So the guardrail has to be in there. On the other hand, nothing stops you from having the guardrail in two different places, one where it generates and one where you verify.

Matt Coles: 28:17

so may I, may I offer a happy alternative? Um, so keep in mind well, maybe, happy, happy, uh, keep in mind that we should, we should keep in mind that we know that input, input filtering and output sanitization have opportunities for failure and bypass. So I would, uh, suggest, do you want to let the filters take care of the sanitization or do you? So, for instance, let's use a, use a good example here. Do you want the filters, the input filters and the output sanitization to, um, to vo help help cha make it sure. Make sure that a developer can't, uh, uh, inju introduce, um, malware, you know, ransomware into their code or some sort of a crypto miner into their code. Or would you rather have the LLM not know what crypto mining is in the first place? So if you have, if, if you have to whitelist or, you know, or, uh, you know, whatever, do, do filtering on the inputs and outputs, there's opportunity for bypass. If the, if the AI system doesn't know what you're asking, it can't answer you. And so

Chris Romeo: 29:36

just, I'm talking about a I don't care, I don't care if the AI system improves upon the work product. I'm just saying I'm not going to trust the AI system to give me a checkmark at the end of the process that says, Oh, don't worry. It's secure. I have way to verify that because nobody on earth explain how an LLM actually

Matt Coles: 29:56

but you don't have to, but you don't have

Izar Tarandach: 29:58

but you don't the code that comes out is what you

Matt Coles: 30:00

exactly

Chris Romeo: 30:01

saying, but you need another place to check it, you have to

Izar Tarandach: 30:05

but verify.

Chris Romeo: 30:06

you can't, can't, you can't, you have to have another step that says, I got this thing from this box,

Matt Coles: 30:14

We need to put the safety nets back in place.

Chris Romeo: 30:16

yes, yes, it can't put

Izar Tarandach: 30:18

and you do realize that,

Chris Romeo: 30:19

initial box,

Izar Tarandach: 30:20

and you do realize that this is exactly what the human developer goes through. They have the guardrails that we ask them to follow in their heads, they generate their code, and we trust it somewhere else. So we are not proposing anything,

Chris Romeo: 30:34

I can't fire My LLM. sue I can't, I take my LLM to court,

Matt Coles: 30:41

You can plug

Izar Tarandach: 30:41

That I don't know. But you can unplug it and change for a new one. I do it

Chris Romeo: 30:45

can I sue, can I sue it for, can I have criminal charges brought against it for malfeasance, for

Izar Tarandach: 30:52

a lawyer.

Chris Romeo: 30:53

code into my system that resulted in me losing 10 customers worth 10 billion in revenue?

Izar Tarandach: 31:00

If you're getting an LLM as a service,

Chris Romeo: 31:02

sue an LLM, who am I gonna sue?

Matt Coles: 31:06

They're a

Izar Tarandach: 31:06

somebody's giving you a service. Yeah.

Chris Romeo: 31:08

If I'm getting it from a commercial provider, if I'm using a, uh, an open source LLM, there's nobody. It's not a person. I

Matt Coles: 31:16

it's as is. It's used as is. The licensing terms usually are, you know, use it at your own risk.

Chris Romeo: 31:21

That's why, but I'm one of the differences between a human being and an agent is I can

Matt Coles: 31:27

Oh.

Chris Romeo: 31:28

up

Izar Tarandach: 31:28

No.

Chris Romeo: 31:29

criminal charges against a human being who does something on purpose to maliciously

Matt Coles: 31:33

if you use

Izar Tarandach: 31:34

can you one of your developers

Chris Romeo: 31:36

Of course I can. If I needed to. Not, I mean, if it was a, if it was an issue that, let me give you, I mean, not if they made a bug, if they introduced a SQL injection, of course not. But if they were working with a nation state, an outside state, and knowingly injected a piece of malware into my product as an insider threat type of risk scenario,

Izar Tarandach: 32:00

that, you want to sue does it knowingly do it because

Chris Romeo: 32:05

that's hard to know. don't know

Izar Tarandach: 32:08

you want to sue the LLM, then you have to put it, then we have to put it along the same parameters as the human right.

Matt Coles: 32:14

So, I also threw out, can you sue an open source, can you sue an open source project if you take code that's, that's under the MIT or BSD2 clause license?

Chris Romeo: 32:25

you're, you're, you're, you're giving away your, you're indemnifying them as a component of the license.

Matt Coles: 32:30

That's right. And so if they delivered crypto mining software and you consumed it, It's your fault. You didn't proper, proper validation.

Izar Tarandach: 32:39

Yes, correct.

Matt Coles: 32:40

So if you're talking, cause now you're talking liability, right? And liability is a, is a big topic on its own.

Chris Romeo: 32:48

That's where I always land. It's always about liability with me.

Matt Coles: 32:51

Why, why is that? Why is it the corporate leaders are always thinking about liability? Geez.

Chris Romeo: 32:55

All right. Well, we've got couple more, we've got a couple more minutes. I want to, I

Matt Coles: 32:58

well, so I do want to throw. I do actually want to throw out one other mind bending question actually along the

Chris Romeo: 33:03

go first. You

Matt Coles: 33:04

Yeah, really, really quickly. So we're just, we've been talking about inputs and outputs. I will remind, I want to throw out the concept here that not only is it input and output and the developer can take that code and use it, but that developer probably is getting prompted on, on the output side of, oh, you don't like answer one, here's answers two, three, and four. Which one do you is best? Right? And that provides a couple things. It provides an alternative code, so now we have potentially multiple choices of code that could have been introduced, and which one did they choose, which ones did they have options. You don't see all the options when the code is checked in and validated at the, at the build time. So there's, there's that interesting aspect as well as that becomes training data for the, the, the, the data set of who chooses what we talked about this with the, the HackerProf is human, human augmented training,

Chris Romeo: 34:02

human augmented, but also human augmented AI. It's not, in that case, it's not the LLM giving you the final answer. It's a human, which is really where I think AI has the most benefit in the next five years is by making me a better security person, making you a better developer, if you may, if I make you a 20 percent better developer, That's a, that's a good, that's a good proposition.

Matt Coles: 34:26

Now, wouldn't it be interesting if you take those three, those three or four answers that come back from LL, from, from AI version, AI, you know, AI one, and you pass it to AI two to help the size, which one of these is, is more secure? Yeah. me understand

Chris Romeo: 34:41

for me, no, because what, what, what I needed to, what we need to achieve as an industry for me to be happy with these systems talking to each other is some way, how do we measure trust in this from this solution that I'm getting?

Matt Coles: 34:57

How do you measure

Izar Tarandach: 34:57

do you measure trust with people?

Matt Coles: 34:59

I mean, exactly a problem.

Chris Romeo: 35:01

I mean, it's a good question. I mean, when I hire somebody, I do a background check. I ask for references. I, I build a body of experience or knowledge about that particular individual that is If you like, and if I find out, oh, you guys knew this person and I ask you and you guys are, and you don't vouch for them, that would tell me I trust you guys. So there's an inherent trust in the network effect that I don't, but I don't have any trust in a model today. If you tell me the model is going to generate code for me and I ask you, how can, how can I measure, how can I gain it? How can I answer the question of, do I trust it or not? I'm, I don't think there's a way to do that today.

Izar Tarandach: 35:45

I'm going to offer something that I think that we have explored in previous episodes. We should be treating AI and LLMs as a junior programmer and putting exactly the same apparatus that we have to build this trust on top of it. We should we should, we should actively fight,

Chris Romeo: 36:10

me your take on how you work with a junior, like, what are you saying? Like, what are the constraints you would put on a junior human developer?

Izar Tarandach: 36:16

before going there, we should actively fight this tendency that we seem to have, to think that only because the code comes from an AI, it's better code, and the AI is a god coder. We, we are all very lucky to know people who we could classify as god coders, and their code is nothing like what I see coming from AIs.

Chris Romeo: 36:40

Yeah, and

Izar Tarandach: 36:40

to answer your

Chris Romeo: 36:41

back trust. But yeah, so what does, what a junior developer give me though? Like, what are, what are my, what are my constraints I put on a junior developer that we could then apply to an AI agent?

Izar Tarandach: 36:50

everything that we ask on the SDLC, we should be asking from code coming from an AI, without jumping any hoops, without giving it any passes just because it's from the AI,

Chris Romeo: 37:02

Okay, so SAST, SAST,

Matt Coles: 37:06

the safety

Izar Tarandach: 37:06

SAST-icize that thing!

Matt Coles: 37:08

all the safety, all this, the safety net that we talked about earlier that we've

Chris Romeo: 37:12

SASTicize, we're gonna, we're gonna SASTicize it, we're gonna SCATicize it for

Izar Tarandach: 37:17

We're not going to DAST icize it if we

Chris Romeo: 37:19

going to DAST icize we've already talked about that ad nauseum. What about review? So are you doing

Izar Tarandach: 37:25

Yes, definitely, definitely,

Chris Romeo: 37:27

So senior developer is doing a code review of what's coming out of the AI agent as if it was received from a junior developer. Okay?

Matt Coles: 37:36

The PR process. as before,

Izar Tarandach: 37:40

and here in my, in my, my opinion comes the, the, the biggest part of the thing, the most important bit that I would require. I never require from a junior, from a junior developer a full system in one go, a full functionality in one go. I require from them small snippets of code that over time and can build the trust that they know what they're doing. So that the PRs of that code can be done

Matt Coles: 38:10

even,

Izar Tarandach: 38:11

focusedly. And functionality, I would never ask a general developer to write me. Critical functionality.

Chris Romeo: 38:19

So your authorization and access control policy enforcing component, you're not going to ask a junior dev to build. True.

Matt Coles: 38:27

but there's something missing in this conversation still. We're talking about trust upfront. Meaning, do you trust this developer to, to be reputable, to, to be capable of generating code that is useful in this system? We're what we really are talking, what really need to be talking about, what is our side, I think we started with was, do we trust the output and w. Whether it's a junior developer or a senior developer, we're still going to run through all of those trusting the output steps. Just because a senior developer writes code and commits it to the code base doesn't mean we don't run SAST on it.

Chris Romeo: 39:03

True.

Matt Coles: 39:04

So none of that should change. If it was AI generate code, if it was AI provided code to a developer, it should go through all the same processes.

Izar Tarandach: 39:13

Yes.

Matt Coles: 39:14

And so, do we actually need to establish trust ahead of time if we have the ability to trust the code output,

Izar Tarandach: 39:24

Well, I do, but for different reasons, not for the code itself, for the fact that, that developer is going to have access to a number of other stuff beforehand, so that there's that trust, but that's a separate thing, I think.

Chris Romeo: 39:36

have a

Izar Tarandach: 39:37

you're right, for an AI, why?

Chris Romeo: 39:40

logic problem in that all of the tools we have in our SDL today are not business logic. Like, SAST doesn't find business logic problems. Um,

Matt Coles: 39:51

a human code review can.

Chris Romeo: 39:52

human code review, yeah, that's true, that's true.

Izar Tarandach: 39:54

and there's another tool,

Chris Romeo: 39:57

It's, the, the, always the challenge with human code review is the code review is only as good as the human. And so some,

Izar Tarandach: 40:05

code

Chris Romeo: 40:05

are at code review than others.

Izar Tarandach: 40:08

the code reveal is only as good as a human. equipped with a good threat model.

Chris Romeo: 40:14

Yes.

Matt Coles: 40:16

Alternatively, a lot of math. So, recall the latest,

Izar Tarandach: 40:21

here we go, formal proofs

Chris Romeo: 40:22

methods. No, wah, wah,

Izar Tarandach: 40:25

Jar, jar, jar.

Matt Coles: 40:29

How can you not bring it up on the topic?

Chris Romeo: 40:32

How

Izar Tarandach: 40:32

I don't have a tuxedo.

Matt Coles: 40:35

So so, you know, the, the, the title of this episode might, might be, uh, how, how to, how to, um, get over, uh, human, human bias against AI enabled developers.

Chris Romeo: 40:49

It really rolls off the tongue

Matt Coles: 40:51

Well, cause we're, we're biased. We have a bias against

Chris Romeo: 40:53

Apparently. Yeah, apparently we, apparently I do. I know. I, I apparently I'm the worst of this group.

Izar Tarandach: 41:00

Wait, wait, I have to check something here.

Chris Romeo: 41:02

Oh boy. These are now going to various AI agents.

Izar Tarandach: 41:07

No, how I learned to stop worrying and love the AI. Ha ha ha

Chris Romeo: 41:14

title. And with that, we are out of time for this episode of the security table. This was a mind expanding conversation. Like I did not come into this with a lot of preconceived notions. I was kind of developing them on the fly as we were, as we were sparring back and forth. But this is a, this is a challenging issue. We got to deal with these things philosophically. It's, it's more than technically it's philosophically and then technically, which is really the way we need to do this. So this will not be our last conversation about AI on the security table. That,

Izar Tarandach: 41:45

treat AIs as junior developers, do they get the rights of junior developers? Do AI have rights?

Chris Romeo: 41:51

yeah, they get PTO

Izar Tarandach: 41:53

Philip K. Dick when I need him?

Chris Romeo: 41:58

Do Androids Dream of Electric Sheep?

Matt Coles: 42:00

Electric sheep.

Chris Romeo: 42:02

that, we'll leave, uh, we'll leave you, uh, at the end of this episode. Thanks for tuning into security table and we'll be back in the future to talk more about AI.