Bug Bounty Theater and Responsible Bug Bounty

Izar, Matt, and Chris discuss the effectiveness of bug bounty programs and delve into topics such as scoping challenges, the ethical considerations of selling exploits, and whether it is all just bug bounty theater. The hosts share their insights and opinions on the subject, providing a thought-provoking discussion on the current state of bug bounties in the security industry.

FOLLOW OUR SOCIAL MEDIA:

➜ X: @SecTablePodcast
➜ LinkedIn: The Security Table Podcast
➜ YouTube: The Security Table YouTube Channel

Thanks for Listening!

Transcript

Copy Transcript Copied

Chris Romeo 0:09

Hey folks, welcome to another episode of security table where I thought we were going to have a new member of the table. Uh, even though there was a, literally a dog sitting here a second ago. And so welcome to the second half of what's already been a 30 minute conversation behind the scenes about lots

Izar Tarandach 0:25

Under the table.

Chris Romeo 0:26

and under table. Yeah, that was, that's our new podcast. It's called under the table. Uh, we, we, we use a drinking game format. And, uh, based on various, various four letter acronyms, like if someone says, you know, DAST, it's like drink your whatever you have in front of you. SBOM is just a single shot. There's

Matt Coles 0:44

Whatever happened to RTFM? Come on.

Chris Romeo 0:49

right. But we do have a serious topic that we want to talk about today, and that is bug bounty. And is it still useful in the year 2024? Or 2025, depending on when you're listening to this. So bug bounty, let's, let's start by just laying out, what do we think? When we say bug bounty, what do we think it is? And I've already talked enough. So wants to define bug, like what is a bug bounty?

Matt Coles 1:14

What is a Bug Bounty? So Bug Bounty is, uh, simply a company makes something available for Security researchers to look at provides an opportunity for security, reachers researchers to do legitimate work to find issues and report them responsibly for some sort of payment, right? They don't do it off at the, they don't do it for their civic responsibility from our last episode. Uh, they don't do it for the goodness of their heart. They do it for swag and a leaderboard and, and finance and money. Uh, so it's basically paid QA. for security issues.

Chris Romeo 1:55

Okay. So that leads me to think right off the top is bug bounty cheaper for less expensive for an organization versus what they may have done previously. Like is bug bounty. preventing or, or pushing off pen testing. So we're doing less pen testing because we have a bug bounty. And is that bug bounty costing less than what maybe we did in the past? Cause I've heard some horror stories about people in the bug bounty, like researchers that aren't making very much money that are, they're doing a lot of work and they're getting paid. Like, over time, what they're getting paid is going down more and more, like, it's not like when Bug Bounty first started when there was a lot of hype and there was a lot of money being thrown around, like, it's become not as lucrative. As an opportunity, whereas the top level people make lots of money, but there's lots of people that are spending a lot of time that are not making very much money in their, in their endeavors. So, what do you guys think about that?

Izar Tarandach 2:56

Wow, a lot in that.

Matt Coles 2:58

yeah, so I think maybe,

Chris Romeo 3:00

Unpack it!

Matt Coles 3:01

so, well, actually, let me just, I'll just throw in a, a, a, a, a possibility here. We have heard over the years that it is cheaper to fill, to fix issues sooner in the development life cycle than later. So right the bat,

Chris Romeo 3:15

study, yep,

Matt Coles 3:16

Well, or, or Veri, and Verico did one and, and yeah, others, others have copied it over the years. And, well, we pitch it because, well, if you want to do threat modeling to find design issues before you build it into your product, because in theory it's cheaper and whatnot. And so, if you hold that view, then a bug bounty is you've already shipped. And so, you've already spent engineering time to build something in that, now you're going give to QA, to, to you, to QA testers who are not your own. to find problems and report them to you that you're then going to pay them for. You're probably going to pay them a lot less than a full time employee, but you're going to get varying quality of results. So is it cheaper, to your point? I think there's an argument to be made that it isn't. Unless you get a high volume of quality results.

Izar Tarandach 4:13

Yeah, so I'm gonna start talking and I have no idea where I'm going to end with

Chris Romeo 4:19

do that every day, so that's, that's a common occurrence for me, but go ahead.

Izar Tarandach 4:22

but, uh, okay, so once upon a time, pen testing, it was a very, um, exclusive thing. You had a very limited number of people who were able to properly, changing values of property, do a pen test. And before standards came up and people started. looking at things in a more industrial kind of way, I think. It was very much a game of chance, what kind of results you would get, because it was natural to expect that any given pen tester would focus more on those things that are more interesting to them or that they know more about. And you could be shown, hey, you have this one thing here, but that didn't say anything about the other huge amount of things that might be there and did not get discovered in a pen test. Now, when you turn that around to bug bounties, then you end up with, at least the scenario that I have seen, perhaps doesn't reflect the whole industry, but that's my personal experience. You end up with people who are very good at a specific thing. and rather than just that one place, they go looking at all the things that offer a bug bounty for that specific thing. it could be said that if your bug bounty is well managed and well put out there, and you invite the right people, or you open it to the public, it could be said that you are getting a certain amount of depth in that specific thing, hoping that there's enough breadth in the number of people who are coming to your bug bounty. that that's going to end up giving you better coverage than a pen test. Of course, by this I'm not talking about those pen test providers that we know and love, who give you like a team and the team is diverse and all that good stuff, but those are expensive beyond limit.

Matt Coles 6:26

So, so along that, along that, I'm going to interrupt you I know you, I know you want to talk and talk and talk and, and, but I want to ask you a quick question on

Izar Tarandach 6:33

Me?

Matt Coles 6:34

Yeah, you.

Izar Tarandach 6:35

I ever done that?

Chris Romeo 6:36

Ha, ha, ha,

Matt Coles 6:38

So, so, because you bring up with pen, with pen testing, right? The big thing about pen testing is scoping. Bug bounties are also scopes. Do you find, I guess in your experience, have you found that bug bounties are. A, properly restricted to, to focus the, the, the ask for the crowdsourcing, yet not too restrictive as to, as to bind that crowdsourcing in terms of the technology type or, or other, other factors you were just talking about.

Izar Tarandach 7:09

So I personally have seen at least two kinds of bug bounties that come to mind. Those that are scoped per property. Hey, here's our thing, go and break it. And those that are scoped by kind of stuff that you want to hear back. We are interested in OWASP top 10, whatever. So, what was the question again?

Matt Coles 7:36

Do you find, do you find that, do you find it overly broad or overly restrictive on average

Izar Tarandach 7:42

Okay, so the, the, the, the over the broad, I, I, I have seen teams that have to keep answering people, especially because there's a lot of spray and pray in that. They have to keep telling people, hey, sorry, thank you for your, for your submission, but that's out of scope. And I'm sure that HackerOne has put some statistics about that kind of thing or whatever. But. On the other hand, yes, sometimes you could, by design, make it so restrictive that you're going to get a limited set of results. Then the question is, are you doing that out of, I don't want to say ignorance, but out of limited resources? Or are you doing that because You think that that's the bit of your process that's lacking and you want more deeper verification on that specific thing. And if that's the case, what do we as practitioners can learn from that focus to improve our processes?

Chris Romeo 8:44

Something you said just made me think of Bug Bounty Theater. Do we have bug bounty theater these where like bug bounty is one of those things it's like expected like if you ask a good sized company Say even 200 people and more in the startup world. Oh, do you have a bug bounty? Yes, it's like one of those checkbox items now, like has one. Um, but I guess is there, is there a, is there a bug bounty theater happening where people really aren't getting results from it, but they're playing along because this is something we have to do now to look like we're serious about security. So they have the program, but they're not paying anything out because they're not getting any quality results. Because they've scoped things in such a way that doesn't allow somebody to just go wide open on their system and break it in any way they possibly can.

Matt Coles 9:35

Or, or they're, or they're scoped it to, Oh, uh, we want to understand our, our sales platform or our, uh, our community support site and not the thing that we're actually selling. Right. So, uh, you know, you may have completely scoped out the thing that people are caring about when they asked you have a bug bounty and yet they still have a bug bounty. So to that point, I guess, is that, is that happening? Is that a prevalent thing?

Chris Romeo 10:00

So I'm gonna, I'm gonna, you know, Bruce Schneier and I just, just went together on this term, Bug Bounty Theater, invented this idea of security theater. And so I'm taking it and we're evolving it one step further into Bug Bounty Theater. Because one of my challenges I've always had with Bug Bounty is companies will start Bug Bounty too early. Like they'll build a new product like we have to, we have to do a bug bounty for this and like they would get like cross site scripting then be paying, you know, five grand or ten grand for a cross site scripting. It's like you could have just done this yourself, like this part of just good hygiene and not having proper, applying proper security engineering processes, principles, components to the thing that you built, you just created something and you threw it to see if it would stick on the wall and you let people start going after it.

Izar Tarandach 10:49

but

Chris Romeo 10:49

that's always been my challenge.

Izar Tarandach 10:50

No, but but those are two different problems. The bug bounty theater is basically the same thing as self attestation. You are telling the world, hey, I have this part of my process. If you do or you don't, I don't care. You can always say it's a limited invitation only thing, and go prove that they don't have one. But, the throw it out and get people to test it for you, that's more of a I could see in certain instances, for example, a very small startup with something that they think that could be high profile, very quick, throw it out as part of testing and be ready to pay for that kind of thing. And I don't know if I would call that theater as much as a conscious transference of responsibility to the world out there. So I'm saying, hey, I know that I have to do testing. I know that right now I'm not equipped to do that well. Unless, you know, you run DAST.

Chris Romeo 11:53

Well,

Izar Tarandach 11:53

but, uh,

Chris Romeo 11:54

what just described though. What you just described is what I refer to as responsible bug bounty.

Izar Tarandach 12:00

Yeah.

Chris Romeo 12:00

Which is the opposite of, of bug bounty theater. So

Izar Tarandach 12:03

but, but that could happen at, at, that could happen at every level. I think that the bug, bounty, bounty theater would be to come up and say, oh, we have this bug bounty, and it's very limited. We only get really, uh, elite, super bug bounty hunters and whatever. By the way, you're not one of them. Don't ask for details because we're not going to give it to you. Rest assured we have good people on it. Right? So that for me would be bug, uh, bug bump, bug bounty theater. But the other one, the other one, I could see that transference of responsibility being done in a logical way.

Chris Romeo 12:36

responsible bug bounty, which that's way bug bounty was supposed to be done. What you just described is a fair assessment of how the bug bounty should work and should be used by a company. Matt disagrees with us, so let's hear

Matt Coles 12:50

I was just as a crowd crowdsourced security testing So how how is that different from hiring a contract a contractor consulting firm to do your testing for you?

Chris Romeo 12:59

Well, you only pay for results, So you pay for a, you hire a contractor, how many times have you seen a pen test report where it comes back and they're like, we didn't find anything? And I'm like, what's wrong with you people? Anybody can find anything in any, like no system is so secure that there is zero findings to be had with it.

Izar Tarandach 13:18

You're not paying for false

Matt Coles 13:19

I'm going to talk about, we can talk about scope. We have to talk about scoping yet again, then if we're going to do that, because that's the pen, the problem with pen testing, and I just, or maybe I'm answering my own question, you know, thank you for hooking this together. Pen testing is usually time and time resource box, whereas bug bounty is open ended to some extent.

Chris Romeo 13:40

But oriented.

Matt Coles 13:41

And results oriented.

Chris Romeo 13:42

Pay for results, basically.

Matt Coles 13:44

But you don't get, maybe you don't get to control it as much, so in other words, you're really relying on somebody, it's not like an Uber, you're not calling an Uber to do testing, you're, you're, you're sort of, uh, bird watchers. If you happen to be out in the field, and you happen to find an issue, and you happen to report it, and you happen to know what you're doing. I'm going to get the results from that. I'm going to pay you as a result, but opposed to, I want to hire you to actually find problems on my schedule.

Izar Tarandach 14:10

One thing that's gets, that doesn't get discussed enough. I think, as usual, when, when it's about looking into the engine. It's the internal side of bug bounties, right? So if you're an nsec team and you have a, a, a bug bounty, and you're using one of the, the bug bounty, uh, manager services out there, every time that you get something, there's a, uh, an interesting discussion around risk, around the, uh, prioritization because the person presenting the bug wants to bring that criticality up as much as they can so that they earn more.

Chris Romeo 14:50

Yeah,

Matt Coles 14:51

So wait, you want, do you want to, do you want to explain to people what the, what the pay, how bug bounties pay out?

Izar Tarandach 14:58

Wait, wait, wait, wait, wait, the middle the road, the maintainer is just trying to like basically play a broken telephone game with a person that knows what they are attacking and what they're getting and a person that knows the inside of the system and their own criticality and severity scale. And the AppSec team is just trying to decide is this something that I have to stop everything now and deal with it? Or is this one more person telling me that my certificate is going to expect to end in six months and I should pay a bounty for them to tell me that? So just having the debug mounted doesn't, well it does, it does take some work off of the AppSec team, but it gives them more work at the same time.

Chris Romeo 15:49

They have triage. They have to triage the results to decide whether, and you brought up another point, this contention between the researcher says, this is a critical, the AppSec team says, this is a low,

Izar Tarandach 16:03

But, but apart from the triage, if you do internal triage, at least it's two, two groups trying to talk about the same thing using the same language. Engineering wants to bring that criticality down as much as they can so that they don't have to deal with it, and AppSec wants to bring it up. as much as they think it should be so that it's dealt in the right time frame. But now when you put the bug bounty into place, now it's three people talking completely different views of the same thing. So there is even much more in that interaction because now you have a translation layer to put on top of the triage.

Chris Romeo 16:42

But those, those kind of operate in a vacuum. From what I've seen, being like, I've not seen a bug bounty where engineering AppSec team are working together on criticality. It's normally the AppSec team that's triaging and then reassigning those tasks for mitigation via engineering resources without their, without them having a say into what the actual criticality is.

Izar Tarandach 17:07

Then it's even worse in that case because now you have Twice the process, once to determine how much the payout is going to be, and once to determine how fast the solution is going to be.

Matt Coles 17:21

Well, so is is the scheme, is the scheme flawed then? So should, should payout be based on criticality? We talked about this, we brought this concept up last, last time, also, uh, last, last session. Well, I guess it hasn't posted yet. So people will see it when they see it, and then it'll be the last one. Uh, that

Izar Tarandach 17:38

won't be last one.

Matt Coles 17:39

It's always be the last one. Um, but they'll, you know, we, we talked about this idea that, um, so if you see this out of order, you won't know what I'm talking about until you see the last one, uh, that. Um, should, should these types of issues, if you're going to submit a flaw, a flaw, a bug, whatever, a vulnerability, or a PR, um, in this case, I guess, a bug bounty submission, should this be, um, should you pay for, for quality, quality of the report, not severity of the issue? In other words, if the bug, if the bug bounty is, if the bounty really is paid and bounty, I get, and you, I mean, it could be again, leaderboard, it could be financial, it could be other things. Um, you know, it maybe it needs to be a, a submission of a problem and a fix or a problem and some sort

Izar Tarandach 18:30

oh, we did talk about that, yeah.

Matt Coles 18:32

Yeah.

Chris Romeo 18:32

I think, you

Matt Coles 18:33

as opposed to severity.

Chris Romeo 18:34

of quality though, like nobody's ever going to pay on quality. Right, because, yeah, but watch what happens. I'll just go write a bunch of quality reports. For low findings and get paid at a higher rate because of my quality metric that they're being paid on, right? It's, it's a pay for performance and the, the most critical things are gonna, are gonna drive the biggest payouts because they're gonna be the things that, if they were zero days that were in the underground. Not known or if a researcher found it and then went the other way because that's the other thing we haven't even touched on yet Right, there's a whole marketplace for security research an underground marketplace Some of it's not underground some of its above ground but like nation states pay for exploits There's a market, there's a, there's a, there's a whole cottage industry of nation states that will buy exploits from security researchers.

Izar Tarandach 19:29

hmm.

Chris Romeo 19:29

So now you've got this dilemma as a researcher between, do I play in the, in the bug bounty space and minimize the amount of money? Or do I take something juicy and try to broker it through the underground?

Izar Tarandach 19:42

I have feeling that we are now talking about the same players.

Matt Coles 19:45

now we're talking about ethics, right? We're talking, we have that ethics conversation

Chris Romeo 19:49

always into everything,

Izar Tarandach 19:50

No, no, no, no, but even before that, even before that, I think that's the people who are in the market for zero days at the national security level. Are not the same people who are running around spraying and praying on bug

Chris Romeo 20:04

bug, Oh yeah, I think they totally are. I think there's same people. I don't think, I think they, why would they not use, if you've got the level of skill to be able to find exploits and sell them to nation States, why not double dip on both sides of the equation?

Matt Coles 20:20

Because he's about,

Izar Tarandach 20:21

Then you go to Pwn2Own and you get a million dollars and you, you go to sleep.

Matt Coles 20:26

But I think you're talking, you're talking about spray and pray, you're talking about what we traditionally have called script kiddies. You run the scripts, you run the, you run the scanner, you run the DAST scripts, whatever, and you get, a result and you report it, which the company could have done themselves, but they didn't. And so they went ahead and

Chris Romeo 20:44

are we, are we drawing a conclusion that the majority of bug bounty work that's happening right now is equated with script kiddiness of the 2000s?

Matt Coles 20:55

No, not, not suggesting that at all. Just, I was just

Chris Romeo 20:59

for controversy.

Matt Coles 21:00

Yeah, I was just commenting, I was commenting on what Izar was saying is spray and, spray and pray is you identify, you create a, it could be a menisploit too, right? It, somebody is running a, a test that either they wrote or somebody else wrote against a whole swath of stuff and submit, submit, submit, submit.

Izar Tarandach 21:18

Just an example. Okay. Somebody who specializes in all forms of, of uh, uh, server side, uh, reference, whatever is sf, SSRF, and then they go from bug bounty to bug bounty looking. only for that, because they know the vectors, they know the ways, they know how to exploit, they know how to, they know that very well, right? So that, that's my spray and pray, it's not somebody, it's not somebody just running a script

Matt Coles 21:45

Oh, okay. Alright. So I misunderstood. I I thought you were, I thought you were talking about scripting. Yeah.

Izar Tarandach 21:49

no, no, at point it becomes, at some point it may become, but not

Chris Romeo 21:54

That's business on their part. That's good business to

Izar Tarandach 21:56

it is, it is.

Chris Romeo 21:57

very good at one thing and then try to find a thousand places I can, I can recoup money on. I mean, hats off to him or her. Nice job.

Izar Tarandach 22:05

until we wake up and figure out that, hey, there is a one shot. way of solving this, do something with a library, do something with settings, whatever, and then that crumbles and they, they move to the next one. But again, it's very smart people, it's very experienced people, and if they focus on that, they have an opportunity to make a decent amount of money. Now just

Matt Coles 22:30

from from legitimate,

Izar Tarandach 22:31

from legitimate words, right? Now to go back to the, the, the zero day thing, just imagine the amount of stuff that's broken in the systems that we are using right now, and we have absolutely no idea what's going on. And somewhere in a dark room somewhere, somebody's like laughing their ass off because they're going over everything that we do every day to protect those things.

Chris Romeo 22:50

mean, that's the difference between a senior security person and anybody else in our field. If you're a senior security person, you know how bad all the

Matt Coles 22:57

know where the bodies are buried.

Chris Romeo 22:59

are around us. You know that this is a, this is a Matrix esque environment we, we, we travel in.

Izar Tarandach 23:07

it's the difference between what keeps you awake at night and what do you mean sleep, right?

Matt Coles 23:13

And, and, and how long and how long is your non-compete agreement?

Chris Romeo 23:17

True.

Izar Tarandach 23:18

what

Chris Romeo 23:19

I mean, eventually you just have to let it go. You just have to realize, I can't, not going to lose sleep over it because it's out my control. And you become, then you become a, is that when you reach a staff level

Izar Tarandach 23:31

No, that, that's the curve, right? You got like 5 percent who don't don't care about it. Everybody panicking in here. And then the 5 percent who said, eh, whatever.

Matt Coles 23:40

Well, and the trick is to watch is the trick is to watch stock stock sales because those senior people usually get stock options start to sell like crazy Covert

Izar Tarandach 23:51

uh, you wanted some controversy. Okay, let's let's see if I can get some. Internal bug bounties.

Matt Coles 23:59

Isn't that just QA?

Chris Romeo 24:00

And do

Matt Coles 24:01

Isn't that just QA testing

Izar Tarandach 24:02

Yeah, but apparently you get paid if you actually find something.

Chris Romeo 24:06

I thought those were gone. I thought people stopped doing internal bug bounties.

Matt Coles 24:10

Hackathons? Are those also called hackathons? Maybe.

Izar Tarandach 24:14

If you're nice about it, I guess.

Chris Romeo 24:16

like you get it you get a t shirt or a coffee mug or something. They're not Nobody's paying cash for those.

Izar Tarandach 24:21

I've seen, I've seen cash.

Chris Romeo 24:23

Really?

Matt Coles 24:24

Really?

Izar Tarandach 24:25

My, my, my question always goes to, it's almost like fiddle on the roof, collusion, collusion, collusion. So

Chris Romeo 24:33

And then I can that's that's that's always been my argument against internal bug bounty It's like why do I want to create an environment where you could introduce something to then fix it and gather a reward? That's just a bad motivation. Like there's better I can I can pay people for doing things like increasing their security knowledge and proving it via a training program. That's what I did at Cisco. We gave a hundred dollars when you finished the first level. And I was happy to send that money. It wasn't my money, but I was happy to send it anybody that did that because it was a positive motivational reward. And there was no way somebody, so yeah, somebody might've tried to game the system and they didn't learn as much. They still had to learn something. They couldn't just go in and they, weren't just, there was no way they could defraud me of the a hundred dollars. Because they had to, they had to learn something to pass the test, right? Whereas in internal bug bounty, I've got this environment where somebody can, somebody can game the system. I don't want to spend time trying to see if people are gaming the system. I got other bigger problems that should keep me up at night, but we've already established they don't. So my next, and now time for my announcement. My next. My next career move is going to be as a CISO because I can sleep night.

Izar Tarandach 25:45

a mattress, of a mattress company.

Chris Romeo 25:49

A mattress company?

Izar Tarandach 25:50

Yeah. Because then you sleep at night and you're testing the product at the...

Chris Romeo 25:54

oh, that would be right there. that's marketing. You're in, you're in marketing I'm going to see if there's a marketing opening at the new company...

Izar Tarandach 26:00

hey, here's where I draw the line.

Chris Romeo 26:04

Izar refuses to, uh, to, to join the, the, the marketing forces. All right. Well, I think that's enough for today. I think, uh, We certainly picked some rocks up, threw them around, looked underneath them, cracked them together, and, uh, somebody's upset at this point.

Matt Coles 26:21

Now our listeners who are listening at the gym are like punching that bag harder or running extra miles.

Izar Tarandach 26:28

In Australia.

Chris Romeo 26:29

True, we're big in Australia, have established that already, so. Hey folks, thanks listening to The Security Table, uh, find a way, if you have other thoughts on Bug Bounty, find a way to get them to us. We'd love to hear what you're, what you're thinking about this, and if you disagree with something, let us know.

Izar Tarandach 26:44

And if you have something that you think that we should discuss ad nauseam, let us know. We're always looking for something interesting.

Chris Romeo 26:50

yeah, yeah, we're always, we're always spending 30 minutes considering what we need to discuss along the way.

Izar Tarandach 26:55

It's Friday!

Chris Romeo 26:57

But that does conclude this episode of Under the Security Table.