How I Learned to Stop Worrying and Love the AI

Dive into the contentious world of AI in software development, where artificial intelligence reshapes coding and application security. We spotlight the surge of AI-generated code and the incorporation of copy-pasted snippets from popular forums, focusing on their impact on code quality, security, and maintainability. The conversation critically examines the diminishing role of traditional quality assurance measures versus the growing reliance on automated tools and AI, highlighting potential compromises between development speed and security integrity.

The discussion broadens to consider the future of software security tools in an AI-dominated era, questioning whether AI-generated code could make static application security testing (SAST) tools obsolete or introduce new challenges requiring more human oversight. The debate intensifies around the trustworthiness of AI in handling complex business logic and security policies without introducing vulnerabilities.

The dialogue concludes by reflecting on the balance between innovation and caution in software development. As AI advances, the conversation centers on ensuring it enhances rather than compromises application security, offering insights, anecdotes, and a dose of humor along the way. Stay tuned for more thought-provoking discussions on the intersection of AI and software security.

Helpful Links:
Article: "New study on coding behavior raises questions about impact of AI on software development" at GeekWire -- https://www.geekwire.com/2024/new-study-on-coding-behavior-raises-questions-about-impact-of-ai-on-software-development/

Follow AI Security Table:

➜ Home: https://securitytable.ai/

➜ X: https://x.com/SecTablePodcast

➜ LinkedIn: https://www.linkedin.com/company/ai-security-table/

➜ YouTube: https://www.youtube.com/@AISecurityTable

Transcript

Copy Transcript Copied

Matt Coles 0:09

Ooh, Oh, this is cool.

Izar Tarandach 0:14

What? What? What did I miss?

Chris Romeo 0:15

it to the world?

Izar Tarandach 0:17

What did I miss?

Chris Romeo 0:18

Go ahead, we're rolling. So let's see, what is the t shirt? Watching the Watchers. Oh, that's,

Izar Tarandach 0:27

Oh,

Chris Romeo 0:27

What was that show, Fringe? It's making me think of the Watchers, but

Matt Coles 0:32

there, it's on the screen.

Chris Romeo 0:33

Yeah,

Izar Tarandach 0:36

yeah,

Matt Coles 0:37

It's a very, uh, it's, it's so not like the EFF.

Izar Tarandach 0:41

yeah,

Matt Coles 0:43

It's very, uh, Illuminati, uh, kind of

Izar Tarandach 0:46

I wouldn't wear that to certain places.

Matt Coles 0:49

I would absolutely wear this to every place.

Izar Tarandach 0:52

No, there are other where you wear that. People say, hey, where did you get that? And you say, oh, I just got it at the last meeting.

Matt Coles 0:59

Like I'm working, I'm wearing that work to work. If I ever have to go into the office, I'll wear that to work.

Chris Romeo 1:05

That's right. Matt is, Matt is all about freedom. He's a big freedom guy. All right, well,

Matt Coles 1:12

electronic freedom.

Chris Romeo 1:13

Electronic Freedom.

Izar Tarandach 1:15

Freedom!

Chris Romeo 1:16

I think, I think you're generally a freedom proponent

Matt Coles 1:19

Yeah, generally, generally.

Chris Romeo 1:20

other than your dogs, which are not allowed to be free and

Matt Coles 1:23

No, that would

Chris Romeo 1:24

roam the

Matt Coles 1:25

wouldn't end well. Because then, because then, I wouldn't be free. What

Chris Romeo 1:35

dogs would be free. All right. Well, on that note, welcome to another episode of The Security Table, where you kind of have joined a conversation in progress, but Sometimes that's okay, because you're getting a behind the scenes view of what happens in the lives of myself, Chris Romeo, Izar Tarandach, and Matt Coles. So, welcome again to, uh, Security Table, where we're gonna focus in on artificial intelligence. A topic that is on many people's minds these days. And no, it's not robots. Izar, Wondered sometimes if is a robot in certain situations. I was like, is this guy a robot? I don't think he's a Terminator because I haven't really seen him like running and turning into metal or running fast or anything. But

Izar Tarandach 2:17

It's worse. Sometimes I wish I were

Chris Romeo 2:20

I mean, we all sometimes wish we were a robot, but that's, that's a story or an episode for a future

Matt Coles 2:27

is it? Domo arigato, Mr. Roboto. I can't, I, we can't use the whole clip, I don't think, that would demonetize, but uh,

Chris Romeo 2:36

yeah, that was a good, uh, that's a good song though. I it's that's seventies, right?

Matt Coles 2:40

70s, late 70s, early 80s. Though, well, late 70s, early 80s, but yeah.

Izar Tarandach 2:45

No, no, no, it's not late 70s. Don't, don't make me that old, please.

Matt Coles 2:48

I have to look. While, while we're talking, I go

Chris Romeo 2:51

I got news for you,

Izar Tarandach 2:52

no, I'm already looking. 1983. Yes,

Chris Romeo 2:59

well, I was just a tiny baby at that point. Well, maybe not, but There we go, this is called an ear wig, which will now be stuck in people that are in Australia and also in the gym at this moment in time.

Izar Tarandach 3:14

it's a great gym song.

Matt Coles 3:17

Mm hmm.

Chris Romeo 3:18

don't know, that really help you to push more weight up in the air?

Izar Tarandach 3:22

Yeah, because gets so peeled that's in your head.

Chris Romeo 3:27

You're so mad. You're like, put more weight on

Izar Tarandach 3:30

just like,

Chris Romeo 3:32

me go for a little bit more weight. is a song's driving me batty.

Izar Tarandach 3:37

me more weight.

Chris Romeo 3:41

People are listening to this going, is this an actual show or are these guys just their

Matt Coles 3:48

Did they hit record too early and they don't know that they're on air?

Chris Romeo 3:51

Yeah,

Izar Tarandach 3:52

it's Friday. We're allowed to lose our minds.

Chris Romeo 3:54

that's true. That's true. Well, let's get out. Let's get ourselves on, uh, on, I can't even think of the word track.

Matt Coles 4:00

gonna lose our minds completely here.

Chris Romeo 4:02

Yeah, I need to, you know, I need, I need to get my inner, uh, my inner actor. I have to tap into that which what's, what's my motivation and my role. Oh, angry security person. Got it.

Izar Tarandach 4:13

Done that!

Chris Romeo 4:14

yeah, my normal daily routine. But so we want to talk about AI. and we specifically want to talk about this study that was, that's recently come out. Um, we'll put a link to it in the show notes, uh, the actual description of the study. The title of it is new study on coding behavior raises questions about impact of AI on software development. And, uh, a couple of notes of things that I kind of took away from this article just to set the stage. Uh, they looked at 153 million change lines of code. Comparing changes done in 2023 versus previous years, uh, before AI hit the scene here. Um, they, they found that code churn or percentage of lines thrown out less than two weeks after being authored is on the rise. This is expected to double in 2024. So people are putting code in and then it's getting ripped out. Percentage of copy pasted code is increasing faster than updated, deleted, or moved code. Their conclusion is this is in response to AI generated code, which is then being pasted in. And then really they kind of summarized it as AI coding assistants are very good at adding code, but they can cause AI induced tech debt. So there's a whole other new, uh, new kind of a term there. So I guess,

Matt Coles 5:35

Yeah, and I wonder, so just to jump right into the deep end on that one, is it, is it technical debt, like we would traditionally consider technical debt of bugs, defects, you know, things that have gone wrong, that would require either redesign or rework later, um, you know, refactoring later. But, but I think what we also have been starting to, at least I've been starting to hear about is, um, uh, things like copyright. copyright infringed code, right? So it isn't just technical debt, it's potentially liability debt

Chris Romeo 6:15

why would that be? Because it's coming from an AI that's been trained on something copyright and then it's passing through and getting pasted. Is that, is that the

Matt Coles 6:23

Yeah. Yeah. So, I mean, I, I can imagine, I, I think if I remember correctly, you know, GitHub, you know, public, public GitHub sources and, and, or public repositories for source code may have been part of training sets. Some of that might be under GPL v3.

Chris Romeo 6:40

Hmm.

Matt Coles 6:40

And now you take that code snippet and you drop it into somebody else's code base. And now that, that comes along for the ride. Right. And this is the stack overflow problem, right? Also the developers even before AI, right? Developers would go to the stack overflow or other popular coding sites and say, Hey, how do I do this? Oh, here's the code snippet. Nevermind there's a license attached to that code.

Chris Romeo 7:00

Yeah. And, uh, and also the fact that that code was likely horribly, horribly insecure, which is the, what we think of as the Stack Overflow problem. But then you got to wonder who is training their AIs from Stack Overflow, because find me a bigger, more prevalent, database of coding problems and solutions on earth. I can't think of one that's bigger than Stack Overflow's database of, of all of the issues that they've tracked over the years. But if you're taking that and feeding it into a model and using it as training data, we know from various, there's been academic studies that have shown this, that, uh, they've profiled lots of different issues there. And often the least, one of the least insecure options is the one that gets, Upvoted the most, and that there there'll be a secure option down at the bottom that's gotten three up votes. And the insecure easy one has gotten 10,000 up votes. And so if that's your training, debt training set, you could potentially, that could potentially be what, what are some of these challenges that this other study's finding?

Izar Tarandach 8:08

So before I, I put my foot in my mouth, we're talking about code that's AI generated right now, right?

Chris Romeo 8:16

Yes.

Izar Tarandach 8:17

Okay, so Stack Overflow, the problem with Stack Overflow is that, as you said, Chris, we get snippets. My problem with AI generated code is that all of a sudden we are getting pages and pages and pages.

Matt Coles 8:32

Big snippets, big snippets, very big

Izar Tarandach 8:33

Big snippets, very big snippets, the biggest snippets ever. And the thing is that in the past few years I have experienced a number of times, uh, developers moving from a monolith to a service based architecture. And what happens many times is that pieces of code that people look at them and say I don't understand this because Joe who wrote it is not, has not been with the company for a number of years. And when I cross those two streams, what I guess is, what I get is that now we are building from scratch pieces of code that nobody understands because Joe that wrote it is an AI.

Chris Romeo 9:16

Hmm

Matt Coles 9:17

And it's no longer here.

Izar Tarandach 9:19

And no longer here. And if I ask it to do the same thing again and explain to me, it's going to do a completely different piece of code that probably does the same thing, but won't be able to explain that one.

Chris Romeo 9:29

So there's a lack of predictability that AI provides, which you're saying is causing a challenge in developing software with ai.

Matt Coles 9:37

Repeatability. Repeatability, probably more so than predictability, right?

Chris Romeo 9:42

Okay,

Izar Tarandach 9:43

it's,

Chris Romeo 9:43

yeah.

Izar Tarandach 9:45

I think that it's even worse than that. Uh, remember 10, 20 years ago the thing with pair programming? think that it got too far, but

Matt Coles 9:57

that was the extreme programming, uh, that was an extreme programming method, right? So

Izar Tarandach 10:01

right, it was a thing for a while. I remember that I went to to check out a place in San Francisco and they had like this long tables, and no computers on them, and people were supposed to come and two people per laptop and work like that. Which is nice, hey, it's two people talking over code, right, and discussing every single line, and right here, right there, give me the mouse, give me the keyboard, no, I'm going to close this thing and drop it in your head. Very, uh, very interesting, uh, debate, but now we are trying to do the same thing with AI copilots and whatnots. How much of a conversation can you have with a thing that's supposed to have all the knowledge ever created about writing lines of code, ever? And how good are you when you are talking to this thing, even in natural language? And I'll be the first one to say, sometimes I find it very, very difficult to express to other people in natural language what I want them to do. So to a machine, it's probably even harder. I don't know, I'm very It's an interesting position to be because from one side, I want the technology to work because A, it's wow, so cool. B, while there's so much promise in here, both in time, even in security and all that good stuff. But on the other hand, I'm like, we had difficulties dealing with software when people actually understood what they wrote and how it worked. Now we are going to drop a lot of agency onto something that we basically don't understand exactly how it works, but we like the result so we're going to keep using this thing. And

Matt Coles 11:46

the study highlights actually, I think what, the one, the point is the points you just raised. So the study calls out specifically copy pasted code being treated as. As if it was a short term developer, right? So it's somebody who comes in, you know, we talk about this all the time, the parachute consultant, you know, the consultant who parachutes in, looks at something, goes, Oh yeah, here's your problem, drops a block of code in and leaves,

Izar Tarandach 12:12

the C go,

Matt Coles 12:13

And then, and then you have to pick up the pieces because you either have to integrate it and figure out how to make it work, or it it's now part of your code base. And then later when you have to go support it, well, what do you do? You have to have, does the, do the developers who are working on the project know what, what, what they're supporting, uh, at this point. And, and I think in nowadays, especially, they're going to turn to maybe another AI system to say, summarize this for me. Or help me interpret this.

Chris Romeo 12:49

so we

Izar Tarandach 12:49

then going to get,

Chris Romeo 12:51

now gonna say, we could be, we could be facing a world where, based on what both of you are describing here, software is so complicated that only the AI can parse it. Understand what it's doing, and then we get it back into that trust problem that we've talked about. It seems like every time we talk about ai, we end up coming back around to the trust issue, and that's really the trust issue if only the AI can understand the convoluted code. But the, the counter to that is we always had that. We, we've always had that time in our career where somebody writes some code and we're like, you look at it and you're like, I don't even know how this. Compiles,

Izar Tarandach 13:32

that's true,

Chris Romeo 13:33

less

Izar Tarandach 13:33

and that, that's true, and

Chris Romeo 13:35

does it. And it does the job. Cause I know I wrote a piece of code of this and in security journeys platform many years ago in the beginning, it was an XML parser. It was a disaster. wrote this thing

Matt Coles 13:45

Well, that was your first, that was your first mistake right

Chris Romeo 13:47

Yeah, but I wrote it and like, it was, and it's, it just worked. And, and the other developers are like, uh, how does, how does that work? I'm like, ah, it works. It just, I did it

Izar Tarandach 13:57

my problem. My problem is not even the complexity, it's the corollary of, uh, I don't know how it works, but it works. The next sentence is the one that bothers me. Don't touch it. Because if you touch it, it's probably going to stop working, so don't touch it. We don't know how it works. But, no, no, you go.

Matt Coles 14:17

Well, I just wanted to jump in with a thought, just a question, sort of a random thought came to head. What's different here? So. We expect developers to make mistakes, right? And we expect developers to have certain behaviors like, Oh, I wrote the piece of code, I don't know how, I don't know exactly what happened there, it compiled, don't touch it, it's fine. We expect those behaviors of developers, of humans. We now have humans who are relying on other developers who are not developers, they're AI, or we're relying on AI to do things as developers. But what's different here? Is it because it's an AI system? Is it because of that lack of trust? Is it that lack of predictability or repeatability? These, these parameters, these things that, um, that we don't understand, uh, or, or have, have questions about. And then I would also tie that to, uh, so when we were talking about this episode and, you know, getting ready for it, I was talking about sort of the, the, the, The introduction of code and then the validation of that code. And so we're, we're sort of at a confluence here where we've, not only do we have developers, we have developers introducing code from other places that are not theirs, uh, you know, AI generated, or we have developers who are not developer, not humans, they are AI, but we've also over the, to get to this point, remove the safety nets. From the development process, right? We've taken out QA, we've taken out, uh, checks and balances in that process, other than code check in to some sort of analysis tool, to a, to a PR, you know, automation.

Chris Romeo 15:58

But that's not, that's not AI specific, right? We've

Matt Coles 16:01

That's not AI

Chris Romeo 16:01

DevOps, DevOps has done to

Matt Coles 16:03

ex Exactly, right. So we've, we've taken away the safety net and the checks and balances that would allow us to establish trust and assurance. Let's go back to security terms here from whether it's developers or machines introducing code into the code base. And, and, and by the way, we're talking about code here, but it could be in introducing components. Hey, I'm going to pull component X or Y or Z out of the, out of the air and drop it here into this build. Right. Um, or, or even The components that are used by the build system, right? All of these things are probably in scope for this discussion. We remove all the safety nets.

Chris Romeo 16:37

I want to, I want to just, want to throw a big question on the table here. And Izar, did you want to finish a thought on this particular, what Matt

Izar Tarandach 16:45

Yeah, no,

Chris Romeo 16:46

I want I want to go in a different direction, but I think I want, I want to throw a big, a big problem on the table.

Izar Tarandach 16:51

so wait, I, I just wanted to, to complete Matt's thought with something that we wrote in the, the fish book. that developers move at the speed of innovation, but security moves at the speed of caution. And I think that that's what we are showing here. They want to run forward as quick as possible, get to the shiny thing in the hands of the consumers as soon as possible. And we are pulling back and putting back and putting back. And again, we are not saying don't do it. We are saying do it, but exercise caution.

Matt Coles 17:25

We're sort of like the, we're the, we're the, we're the restraint, we're the restraint, um, cable on an aircraft carrier. We want the plane to land, but we want it to not fall off the edge in the process.

Izar Tarandach 17:37

so that's not a good analogy, if the cable fails, you expect to happen is the plane to open the throttles to afterburner and just keep going out of the other side.

Matt Coles 17:47

I was gonna,

Izar Tarandach 17:48

don't that that's what we

Matt Coles 17:49

I was gonna say the, it's the parachutes, it's the parachutes on the space shuttle when it lands, but since the space shuttle hasn't flying recently, that's, that was hard. Uh,

Chris Romeo 17:57

so

Matt Coles 17:58

your

Chris Romeo 17:59

my, I want to throw this issue on the table now. It's going to bring together a few things that we talked about. How do guardrails and paved roads differ in an AI developer world? Or do they differ? Do we, do we institute different guardrails? Matt kind of started me on this thread with his thought and then I just started thinking about, I've been thinking about guardrails and paved roads a lot lately. How are we, how are they different or are they different when we have AI software engineers that are not humans?

Matt Coles 18:32

So I think there is absolutely an opportunity here for establishing guardrails and paved roads. So one of the problems we're talking about is, uh, developers who, Either willfully ignore or don't understand, you know, their, their, maybe if they work for a company that many of the companies will have policies about where you get, where you get your information from, or what tools you're using, or if you're an open source developer, you just don't have the time or you or whatever, and you don't, you know, teenager in their basement working on some piece of code, they're going to go out to it ChatGPT or some random LLM or whatever, you know, BARD or Gemini or whatever it is these days, put something in, get some code out, be done. Guardrails and paved roads, we have an opportunity to have, say, a very specifically trained model for that same AI system where, uh, maybe that's based on coding conventions of the, of the company involved. Right, so now, instead of getting random code, you're now getting code that at least thought, maybe it's random code, but now fits in a certain pattern. Or maybe it comes with validation checks, right? And so, go ahead Izar,

Izar Tarandach 19:49

No, no, no, no, no, no, you keep

Matt Coles 19:50

no, no,

Izar Tarandach 19:52

no, you're going exactly where I want you to go. Go.

Matt Coles 19:54

so we established the guardrails by, by, by making, making it possible for developers to still do what they think that they need to do in this case. Don't let the AI just return arbitrary answers. If possible.

Izar Tarandach 20:12

Yeah, and that's great. That's one interpretation. I'm going to offer one that's slightly different. One of the big problems that we have with guardrails and paved roads today is verification. We ask people do it like this, don't do it like that, but we stumble on the verification many times. Now, Matt touched on, on, on,

Matt Coles 20:40

Icon too. looked rumbled

Chris Romeo 20:43

Uh, we're gonna have to hang up this call right now because I think the AI has perhaps taken over the world and I'm slightly

Izar Tarandach 20:49

is it Skynet Day? So the,

Matt Coles 20:53

This this is, this is just mundane Bluetooth problems in a smart speaker that didn't how, didn't know that was turned off at the

Chris Romeo 21:00

all right, Skynet, Skynet, stand down, Skynet, we can continue the episode, Skynet is not in effect, okay.

Izar Tarandach 21:06

so the, the, the place that I was going with this was that A while ago we talked about pair programming and the difficulty explaining yourself to another person and now explaining yourself to an LLM. So basically, using the right prompt, right? Now, Matt touched on the guardrails existing as part, as part of the model, if I understood you right, or part of the framework that's actually creating the, the code. And here I'm going to, uh, exhibit my proud ignorance of, uh, internals of LLMs, and please, if we happen to have any kind of Listener out there that knows this, please, please, please feel free to correct me. I will welcome that very much. The things that I don't understand, I don't think that the guardrails actually get built at that stage. The guardrails would be built either in form of a prompt that we would give before the prompt, like an internal prompt that happens before the prompt that you actually ask the problem. So it's like this preamble to the prompt that becomes part of the prompt. So for every request of, uh, writing code, make sure that you don't introduce a buffer before. Okay. So that thing exists

Matt Coles 22:21

Yeah. Pre, pre, pre promptt. Pre promptt. I,

Izar Tarandach 22:23

Then comes your prompt.

Chris Romeo 22:24

You're putting, but you're putting the trust, you're putting the security control inside of the brain of the

Izar Tarandach 22:35

Yes. That's where I'm coming. That's where I'm coming. where I'm

Chris Romeo 22:38

we would never do that in, we would never give the developer in the real world the ability to say, uh, you just write on a piece of paper whether you think you did a good job and it was secure. or no, we we, we would, we measure it ourselves.

Izar Tarandach 22:51

but, but look at it now measure. Okay. Keep this one on the side for a second.

Matt Coles 22:55

oh, hold, hold. Hold time, time off for just a sec. Chris, have you never heard of the term attestation? Self-attestation?

Izar Tarandach 23:09

open!

Chris Romeo 23:10

Yes.

Matt Coles 23:15

That was me pulling a pin out and throwing a hand grenade, or pulling pin and throwing the pin, pulling the pin and then leaving the hand grenade. I'm not sure which.

Izar Tarandach 23:23

wow, wow. Let's talk legislation. No, let's not. Anyway, so the way that I understand the thing, and a while ago there was a bit brouhaha about people being able to extract from ChatGPT, I think, the whole pre prompt, the whole preamble to the prompts, and end. There's a lot of stuff in there, but for example, a lot of, uh, I'm going to put it in quotes because I don't want to go in there. The censorship that's put on top of the results that come from ChatGPT exists in that prompt, right? So to the issue of controls, there are already some controls that live in that pre prompt. Because people are not supposed to have access to that, are not supposed to have ways of changing that. Prompt injection aside, but that's not a security problem as I say. My point is exactly what what Chris said of the measurement. If things keep going the way that they are going At some point somebody's going to wake up and say, why do I need a SAST? The AI is going to create code that's already SAST-ticized. Why do I need, uh, uh, uh, uh, uh, uh

Chris Romeo 24:29

a term? Sasticized?

Izar Tarandach 24:32

when do sanitization, that's when you do sanitization with SAST. You saw it first here, I'll own that

Chris Romeo 24:38

Love it.

Matt Coles 24:39

that, that sounds like coat cancer. Well, well,

Chris Romeo 24:44

I

Izar Tarandach 24:46

You wouldn't be too wrong. But anyway, or people are going to say, Hey, you know what? We don't need SCA anymore because the AI is going to only suggest libraries and packages that are actually good. So these industries are going to go up. These two industries are going to go down and is happen somewhere else. And what

Chris Romeo 25:04

that's That's such a terrible scenario though, because you're putting all of your trust into something that you can't even explain. Like, guardrails exist because developers are gonna, we want to give them the freedom to do cool stuff and innovative stuff, but they gotta stay within the boundaries of these two markers. You can't go off the side of the mountain with your innovation. How, that,

Izar Tarandach 25:27

are

Chris Romeo 25:27

described is not, doesn't, there's no control there. It's all, you're putting all of your trust into the model, into the

Izar Tarandach 25:33

And you are, no, no, wait, wait, wait. totally right. But have we ever seen people do something like that? Hmm. Hmm.

Matt Coles 25:42

so I'll remind you, remind you both that we had somebody on who talked about, uh, this exact topic when it came to things like, uh, AI being used for military applications, as an

Izar Tarandach 25:54

Yeah, true,

Matt Coles 25:55

So we had a Hack-A-Prompt, Hack-A-Prompt guy, right, who was on, um, great episode. If people haven't listened to they should definitely go listen to it. And, uh, Uh, just a point of correction, I think, Izar, what he highlighted was there was both pre injection, pre prompt filtering and post output filtering, so they do both, uh, and, and in, in the, in that chain, there's a chain, potentially a chain of Uh, input filtering so that you don't get prompt injection, um, and then you get the, the actual prompt execution through the, through the model. And then you have output filtering and potentially validation through another AI system before it comes back to the user. so yeah, go ahead. If you're,

Izar Tarandach 26:39

what I'm hearing from you is that in between the prompt and the actual generation of the code that's coming back to me, and then in between that generation and the code that's coming back me, there's a number of different, uh, uh, filters that

Matt Coles 26:51

there

Izar Tarandach 26:51

no control.

Matt Coles 26:52

there might be. Well, then maybe that's the guardrail though, right? So the, the, and the Pandora, the paved road. So,

Izar Tarandach 26:58

good and a

Chris Romeo 26:58

can't, it can't exist in the same. I can't, I can't accept the fact that the guardrail lives in the same system that the decision and the things are being generated from.

Matt Coles 27:09

well, so why does it have to? So we're talking

Chris Romeo 27:12

saying it can't, I'm saying it I can't

Matt Coles 27:14

It shouldn't, it shouldn't,

Chris Romeo 27:15

can't trust something that's a box that I just, I put a request in for code and I get something out. And then it has a checkmark next to it that says, Don't worry, it's secure.

Matt Coles 27:26

Would you feel, would you feel better it, if your pipeline, if your pipeline was, you, you have an IP address that you, if you have IP address that you ask a question to, and then you get, have a different IP address that you get, get the response on, would that make you feel better? I mean,

Izar Tarandach 27:42

no, but, but, wait, wait, wait. But, but, Chris, see, because of the way that we want the guardrails to be incorporated into the design and the implementation of what we're asking for, the guardrails have to be part of that, for lack of a better word. word, taught, that drives the code that's coming back to you. So the guardrail has to be in there. On the other hand, nothing stops you from having the guardrail in two different places, one where it generates and one where you verify.

Matt Coles 28:17

so may I, may I offer a happy alternative? Um, so keep in mind well, maybe, happy, happy, uh, keep in mind that we should, we should keep in mind that we know that input, input filtering and output sanitization have opportunities for failure and bypass. So I would, uh, suggest, do you want to let the filters take care of the sanitization or do you? So, for instance, let's use a, use a good example here. Do you want the filters, the input filters and the output sanitization to, um, to vo help help cha make it sure. Make sure that a developer can't, uh, uh, inju introduce, um, malware, you know, ransomware into their code or some sort of a crypto miner into their code. Or would you rather have the LLM not know what crypto mining is in the first place? So if you have, if, if you have to whitelist or, you know, or, uh, you know, whatever, do, do filtering on the inputs and outputs, there's opportunity for bypass. If the, if the AI system doesn't know what you're asking, it can't answer you. And so

Chris Romeo 29:36

just, I'm talking about a I don't care, I don't care if the AI system improves upon the work product. I'm just saying I'm not going to trust the AI system to give me a checkmark at the end of the process that says, Oh, don't worry. It's secure. I have way to verify that because nobody on earth explain how an LLM actually

Matt Coles 29:56

but you don't have to, but you don't have

Izar Tarandach 29:58

but you don't the code that comes out is what you

Matt Coles 30:00

exactly

Chris Romeo 30:01

saying, but you need another place to check it, you have to

Izar Tarandach 30:05

but verify.

Chris Romeo 30:06

you can't, can't, you can't, you have to have another step that says, I got this thing from this box,

Matt Coles 30:14

We need to put the safety nets back in place.

Chris Romeo 30:16

yes, yes, it can't put

Izar Tarandach 30:18

and you do realize that,

Chris Romeo 30:19

initial box,

Izar Tarandach 30:20

and you do realize that this is exactly what the human developer goes through. They have the guardrails that we ask them to follow in their heads, they generate their code, and we trust it somewhere else. So we are not proposing anything,

Chris Romeo 30:34

I can't fire My LLM. sue I can't, I take my LLM to court,

Matt Coles 30:41

You can plug

Izar Tarandach 30:41

That I don't know. But you can unplug it and change for a new one. I do it

Chris Romeo 30:45

can I sue, can I sue it for, can I have criminal charges brought against it for malfeasance, for

Izar Tarandach 30:52

a lawyer.

Chris Romeo 30:53

code into my system that resulted in me losing 10 customers worth 10 billion in revenue?

Izar Tarandach 31:00

If you're getting an LLM as a service,

Chris Romeo 31:02

sue an LLM, who am I gonna sue?

Matt Coles 31:06

They're a

Izar Tarandach 31:06

somebody's giving you a service. Yeah.

Chris Romeo 31:08

If I'm getting it from a commercial provider, if I'm using a, uh, an open source LLM, there's nobody. It's not a person. I

Matt Coles 31:16

it's as is. It's used as is. The licensing terms usually are, you know, use it at your own risk.

Chris Romeo 31:21

That's why, but I'm one of the differences between a human being and an agent is I can

Matt Coles 31:27

Oh.

Chris Romeo 31:28

up

Izar Tarandach 31:28

No.

Chris Romeo 31:29

criminal charges against a human being who does something on purpose to maliciously

Matt Coles 31:33

if you use

Izar Tarandach 31:34

can you one of your developers

Chris Romeo 31:36

Of course I can. If I needed to. Not, I mean, if it was a, if it was an issue that, let me give you, I mean, not if they made a bug, if they introduced a SQL injection, of course not. But if they were working with a nation state, an outside state, and knowingly injected a piece of malware into my product as an insider threat type of risk scenario,

Izar Tarandach 32:00

that, you want to sue does it knowingly do it because

Chris Romeo 32:05

that's hard to know. don't know

Izar Tarandach 32:08

you want to sue the LLM, then you have to put it, then we have to put it along the same parameters as the human right.

Matt Coles 32:14

So, I also threw out, can you sue an open source, can you sue an open source project if you take code that's, that's under the MIT or BSD2 clause license?

Chris Romeo 32:25

you're, you're, you're, you're giving away your, you're indemnifying them as a component of the license.

Matt Coles 32:30

That's right. And so if they delivered crypto mining software and you consumed it, It's your fault. You didn't proper, proper validation.

Izar Tarandach 32:39

Yes, correct.

Matt Coles 32:40

So if you're talking, cause now you're talking liability, right? And liability is a, is a big topic on its own.

Chris Romeo 32:48

That's where I always land. It's always about liability with me.

Matt Coles 32:51

Why, why is that? Why is it the corporate leaders are always thinking about liability? Geez.

Chris Romeo 32:55

All right. Well, we've got couple more, we've got a couple more minutes. I want to, I

Matt Coles 32:58

well, so I do want to throw. I do actually want to throw out one other mind bending question actually along the

Chris Romeo 33:03

go first. You

Matt Coles 33:04

Yeah, really, really quickly. So we're just, we've been talking about inputs and outputs. I will remind, I want to throw out the concept here that not only is it input and output and the developer can take that code and use it, but that developer probably is getting prompted on, on the output side of, oh, you don't like answer one, here's answers two, three, and four. Which one do you is best? Right? And that provides a couple things. It provides an alternative code, so now we have potentially multiple choices of code that could have been introduced, and which one did they choose, which ones did they have options. You don't see all the options when the code is checked in and validated at the, at the build time. So there's, there's that interesting aspect as well as that becomes training data for the, the, the, the data set of who chooses what we talked about this with the, the HackerProf is human, human augmented training,

Chris Romeo 34:02

human augmented, but also human augmented AI. It's not, in that case, it's not the LLM giving you the final answer. It's a human, which is really where I think AI has the most benefit in the next five years is by making me a better security person, making you a better developer, if you may, if I make you a 20 percent better developer, That's a, that's a good, that's a good proposition.

Matt Coles 34:26

Now, wouldn't it be interesting if you take those three, those three or four answers that come back from LL, from, from AI version, AI, you know, AI one, and you pass it to AI two to help the size, which one of these is, is more secure? Yeah. me understand

Chris Romeo 34:41

for me, no, because what, what, what I needed to, what we need to achieve as an industry for me to be happy with these systems talking to each other is some way, how do we measure trust in this from this solution that I'm getting?

Matt Coles 34:57

How do you measure

Izar Tarandach 34:57

do you measure trust with people?

Matt Coles 34:59

I mean, exactly a problem.

Chris Romeo 35:01

I mean, it's a good question. I mean, when I hire somebody, I do a background check. I ask for references. I, I build a body of experience or knowledge about that particular individual that is If you like, and if I find out, oh, you guys knew this person and I ask you and you guys are, and you don't vouch for them, that would tell me I trust you guys. So there's an inherent trust in the network effect that I don't, but I don't have any trust in a model today. If you tell me the model is going to generate code for me and I ask you, how can, how can I measure, how can I gain it? How can I answer the question of, do I trust it or not? I'm, I don't think there's a way to do that today.

Izar Tarandach 35:45

I'm going to offer something that I think that we have explored in previous episodes. We should be treating AI and LLMs as a junior programmer and putting exactly the same apparatus that we have to build this trust on top of it. We should we should, we should actively fight,

Chris Romeo 36:10

me your take on how you work with a junior, like, what are you saying? Like, what are the constraints you would put on a junior human developer?

Izar Tarandach 36:16

before going there, we should actively fight this tendency that we seem to have, to think that only because the code comes from an AI, it's better code, and the AI is a god coder. We, we are all very lucky to know people who we could classify as god coders, and their code is nothing like what I see coming from AIs.

Chris Romeo 36:40

Yeah, and

Izar Tarandach 36:40

to answer your

Chris Romeo 36:41

back trust. But yeah, so what does, what a junior developer give me though? Like, what are, what are my, what are my constraints I put on a junior developer that we could then apply to an AI agent?

Izar Tarandach 36:50

everything that we ask on the SDLC, we should be asking from code coming from an AI, without jumping any hoops, without giving it any passes just because it's from the AI,

Chris Romeo 37:02

Okay, so SAST, SAST,

Matt Coles 37:06

the safety

Izar Tarandach 37:06

SAST-icize that thing!

Matt Coles 37:08

all the safety, all this, the safety net that we talked about earlier that we've

Chris Romeo 37:12

SASTicize, we're gonna, we're gonna SASTicize it, we're gonna SCATicize it for

Izar Tarandach 37:17

We're not going to DAST icize it if we

Chris Romeo 37:19

going to DAST icize we've already talked about that ad nauseum. What about review? So are you doing

Izar Tarandach 37:25

Yes, definitely, definitely,

Chris Romeo 37:27

So senior developer is doing a code review of what's coming out of the AI agent as if it was received from a junior developer. Okay?

Matt Coles 37:36

The PR process. as before,

Izar Tarandach 37:40

and here in my, in my, my opinion comes the, the, the biggest part of the thing, the most important bit that I would require. I never require from a junior, from a junior developer a full system in one go, a full functionality in one go. I require from them small snippets of code that over time and can build the trust that they know what they're doing. So that the PRs of that code can be done

Matt Coles 38:10

even,

Izar Tarandach 38:11

focusedly. And functionality, I would never ask a general developer to write me. Critical functionality.

Chris Romeo 38:19

So your authorization and access control policy enforcing component, you're not going to ask a junior dev to build. True.

Matt Coles 38:27

but there's something missing in this conversation still. We're talking about trust upfront. Meaning, do you trust this developer to, to be reputable, to, to be capable of generating code that is useful in this system? We're what we really are talking, what really need to be talking about, what is our side, I think we started with was, do we trust the output and w. Whether it's a junior developer or a senior developer, we're still going to run through all of those trusting the output steps. Just because a senior developer writes code and commits it to the code base doesn't mean we don't run SAST on it.

Chris Romeo 39:03

True.

Matt Coles 39:04

So none of that should change. If it was AI generate code, if it was AI provided code to a developer, it should go through all the same processes.

Izar Tarandach 39:13

Yes.

Matt Coles 39:14

And so, do we actually need to establish trust ahead of time if we have the ability to trust the code output,

Izar Tarandach 39:24

Well, I do, but for different reasons, not for the code itself, for the fact that, that developer is going to have access to a number of other stuff beforehand, so that there's that trust, but that's a separate thing, I think.

Chris Romeo 39:36

have a

Izar Tarandach 39:37

you're right, for an AI, why?

Chris Romeo 39:40

logic problem in that all of the tools we have in our SDL today are not business logic. Like, SAST doesn't find business logic problems. Um,

Matt Coles 39:51

a human code review can.

Chris Romeo 39:52

human code review, yeah, that's true, that's true.

Izar Tarandach 39:54

and there's another tool,

Chris Romeo 39:57

It's, the, the, always the challenge with human code review is the code review is only as good as the human. And so some,

Izar Tarandach 40:05

code

Chris Romeo 40:05

are at code review than others.

Izar Tarandach 40:08

the code reveal is only as good as a human. equipped with a good threat model.

Chris Romeo 40:14

Yes.

Matt Coles 40:16

Alternatively, a lot of math. So, recall the latest,

Izar Tarandach 40:21

here we go, formal proofs

Chris Romeo 40:22

methods. No, wah, wah,

Izar Tarandach 40:25

Jar, jar, jar.

Matt Coles 40:29

How can you not bring it up on the topic?

Chris Romeo 40:32

How

Izar Tarandach 40:32

I don't have a tuxedo.

Matt Coles 40:35

So so, you know, the, the, the title of this episode might, might be, uh, how, how to, how to, um, get over, uh, human, human bias against AI enabled developers.

Chris Romeo 40:49

It really rolls off the tongue

Matt Coles 40:51

Well, cause we're, we're biased. We have a bias against

Chris Romeo 40:53

Apparently. Yeah, apparently we, apparently I do. I know. I, I apparently I'm the worst of this group.

Izar Tarandach 41:00

Wait, wait, I have to check something here.

Chris Romeo 41:02

Oh boy. These are now going to various AI agents.

Izar Tarandach 41:07

No, how I learned to stop worrying and love the AI. Ha ha ha

Chris Romeo 41:14

title. And with that, we are out of time for this episode of the security table. This was a mind expanding conversation. Like I did not come into this with a lot of preconceived notions. I was kind of developing them on the fly as we were, as we were sparring back and forth. But this is a, this is a challenging issue. We got to deal with these things philosophically. It's, it's more than technically it's philosophically and then technically, which is really the way we need to do this. So this will not be our last conversation about AI on the security table. That,

Izar Tarandach 41:45

treat AIs as junior developers, do they get the rights of junior developers? Do AI have rights?

Chris Romeo 41:51

yeah, they get PTO

Izar Tarandach 41:53

Philip K. Dick when I need him?

Chris Romeo 41:58

Do Androids Dream of Electric Sheep?

Matt Coles 42:00

Electric sheep.

Chris Romeo 42:02

that, we'll leave, uh, we'll leave you, uh, at the end of this episode. Thanks for tuning into security table and we'll be back in the future to talk more about AI.