We Don't Know What We Don't Know

Certificate pinning is a security measure used in computer networking and something Chris candidly admits to his lack of understanding.

Matt and Izar explain certificate pinning, a client-side operation that adds an extra layer of security to the Transport Layer Security (TLS) protocol and ensures that the client application checks the server's certificate against a known copy of that certificate.

The discussion leads to a reflection on the vast amount of knowledge required in cybersecurity, emphasizing the importance of continuous learning and the willingness to admit and fill gaps in one's understanding. Engage in further research and discussion, and realize cybersecurity is a "never stop learning" discipline.

FOLLOW OUR SOCIAL MEDIA:

➜Twitter: @SecTablePodcast
➜LinkedIn: The Security Table Podcast
➜YouTube: The Security Table YouTube Channel

Thanks for Listening!

Transcript

ST018- We don't know what we don't know

[00:00:00] ​

[00:00:00] Chris Romeo: Certificate, pinning. Can 

[00:00:11] anybody can anybody really explain certificate pinning to me?

[00:00:16] Matthew Coles: Yes.

[00:00:16] Izar Tarandach: No, first I, first I need somebody to tell me how to say it. If it's pinning your pin. Pinning because it's pinned. Put a 

[00:00:24] pin on it. 

[00:00:24] Matthew Coles: pinning. It's P

[00:00:26] I N N 

[00:00:27] I N G, so it's two, two

[00:00:28] Ns 

[00:00:29] in 

[00:00:29] Chris Romeo: was, if it was pining, you'd be complaining about it, which is what I hear most people do 

[00:00:33] that have to do with it.

[00:00:34] Matthew Coles: Yeah. So if

[00:00:35] you're gonna, if

[00:00:35] you're gonna pine for a certificate that then, then you're lacking certificates, in which case you have a bigger problem,

[00:00:42] Chris Romeo: All right. Well, let's, I'm gonna go ahead and start us right off, even though we're already recording and say welcome to everybody on the security table because you just entered a conversation that the three of us were having. It was supposed to be the preemptory conversation. We weren't planning for it, but we started talking about certificate pending.

[00:00:57] Now I wanna pull in this thread

[00:00:58] Matthew Coles: And actually Chris, if I may before you, before you, before we all came together, Izar and I were having a, a good debate about the newly released CVSS version 4.0,

[00:01:09] Izar Tarandach: Yay.

[00:01:10] Matthew Coles: which is now in public preview, I should say, not fully released as in ga, but in public

[00:01:14] Izar Tarandach: And everybody should read it, and everybody should comment if they feel they need because it's a great piece of work.

[00:01:20] Matthew Coles: absolutely.

[00:01:21] Chris Romeo: Okay. It gets the ESR seal of approval. That doesn't come very easily, folks. That's something that, uh, he only hands out once a decade. So, I mean,

[00:01:30] Izar Tarandach: Wait, when did I say the security journey was good? Uh,

[00:01:36] Matthew Coles: All right. So, so, so retract everything you just said about CVSs, is that what you're.

[00:01:41] Chris Romeo: Nah. Well, he'll, he can use two. I mean, if he really wants to, he can use two. It's fine.

[00:01:45] Izar Tarandach: No, it, it's okay. Security journey was really good. So it counts two levels.

[00:01:51] Chris Romeo: two levels. There we go.

[00:01:52] Matthew Coles: So, so cert pin, so cert pinning,

[00:01:55] Chris Romeo: Yeah, let's talk, let's talk about this. Cause I'm, I'm, I'm, so let me just give a ca uh, give a kind of a, an intro to this because I consider myself to be mildly intelligent and I don't know. I'm gonna, I'm gonna just say, yeah, thank you for your vote there. Your vote of confidence. Uh, I'm gonna say like cert pinning. This is kind of like a, this is a dangerous thing. I'm like, I'm having a, a confessional moment here. I don't know that I've ever really understood what it is, and I think I've, I could explain how browser certificates work and certificate authorities and all the pieces that fit together. But certificate pinning, I don't think I've ever managed to get a solid view of it in my head. And so if you could enlighten me, I will. This will be, this will be a great day.

[00:02:40] Matthew Coles: I'll, I'll, I'll start, uh, if I, if that's okay. Um, so first off, I'll, I'll say the, the, the go-to reference that I use whenever I'm talking with a team about, well, you know, you should look at using cert pinning is there's an OWASP article that that really does a good job of outlining what, what certificate pinning is, what does it mean, what are some of the implications. And so cert pinning is a, an approach, a mechanism for, um, addressing, uh, trust of certificates between, uh, one system to another in a way that the, that the, I guess the, the P k I or, or the, the, the notion of, of a certificate is, is issued by somebody else. Uh, and so you can trust them, so you can trust the cert. There's, there's some challenges with that in your, in certain environments. So, cert pinning really is based on certain pieces of information in a certificate, either in a, a parent to a, to a, to a child, or to a leaf cert, or to the leaf cert itself. Um, and this, this can also work if those, uh, if that leaf cert is a self sign certificate because, uh, obviously it has no parent, uh, or no recognizable parent. Um, That you can decide, uh, to trust the certificate or not, simply because it's something you know about and you can make that explicit decision. So if you're looking at a certificate that's coming from web server, you're going to want, you're gonna, um, look at the information in that cert. Traditionally, you're gonna look at that information.

[00:04:12] That's sort of what your browser does, right? And says, oh, uh, this comes from a trusted signature source, right? The chain I can trust and I can, I think that this is associated with this host because it has the, the O and the ou, et cetera. And that's great if that's not a self-signed cert, if it's a self-signed cert. You don't really have a, a way of trusting it. There's no third party attestation of that trust by, by the signature chain. And so cert pinning provides a, a, a stopgap measure of being able to say, given these specific parameters that I can validate in that certificate, and that may use, that will usually include things like this, the fingerprint of the cert or the issuer of the cert, uh, or, or the ou in case of the issuer will be the same there for softens information on that certificate to say. I trust this specific, I trust this explicitly. So, CERT pending is a way of explicit trust of a certificate, sort of independent of which chain it came from, or potentially including the chain. Um, simply because you may get multiple, a browser may look, may have a cert store that will trust multiple certificates and you really wanna trust one. Not

[00:05:21] any of, if that makes sense.

[00:05:23] Izar Tarandach: Yep.

[00:05:24] Chris Romeo: One and only one

[00:05:25] Matthew Coles: One and

[00:05:25] only one. Or based on this criteria, right, you may, you may pin the one or you may pin to a, an intermediate as you know, the issuer of, so any of the ones that are issued by and, uh, but as opposed to all,

[00:05:38] Izar Tarandach: Which is where things start getting complicated for the developer because then you have, preferably, you have to preload that into the app. So that when you publish the 

[00:05:49] Matthew Coles: yeah, you have to, 

[00:05:49] Izar Tarandach: out with already 

[00:05:50] Matthew Coles: knowledge ahead of time. 

[00:05:52] Izar Tarandach: There, there are some schemes that let you do it at uh uh, the first time that you see it, but then again, you might, depending on the environment that you are in, you might be shooting yourself in the food because the first thing that you're going to see is basically the one that's trying to intercept you.

[00:06:07] So , you.

[00:06:09] Matthew Coles: right. And this is

[00:06:10] where if you had a th if you, if you didn't have a pin already established, you may, you know, for instance, have a call home mechanism. Hey, should I

[00:06:16] trust this? Or gimme my pin list, right.

[00:06:20] Izar Tarandach: But then you start getting more and more complex and complexity introduces, breaks and breaks invalidate the whole 

[00:06:28] Matthew Coles: Yeah. What do you do if you can't? What happens if you

[00:06:30] can't call home your brick? You know, at that point,

[00:06:33] Chris Romeo: So this reminds me of a, a, an episode of the Office

[00:06:37] Izar Tarandach: and as things tend to do.

[00:06:40] Chris Romeo: where Michael Scott is asking Oscar the accountant to explain something to him. And after a while he looks at him and he says, can you explain this to me like I'm five. And then and then Oscar uses this illustration of, well, your mommy and daddy give you $5 to start a lemonade stand.

[00:06:58] And then, um, so I feel like I'm still at that lack of clarity. I don't know that I need you to explain it to me like I'm five, cuz that might be too much. But, so is, is Certificate p a client side operation or a service

[00:07:10] side 

[00:07:10] Izar Tarandach: a client side operation.

[00:07:11] Chris Romeo: both?

[00:07:12] Izar Tarandach: And the thing is that normally when you're verifying, uh, certificates, you are using the normal mechanisms of b k I against some fields and against a, uh, uh, a chain of trust. You, you are. that whomever gave you that key is somebody who's trusted by somebody else and it's trust all the way down.

[00:07:33] Right now on, once you introduced, uh, pinning, you're putting on top of the trust, you are saying, I'm expecting to see a certain certificate that has a certain, uh, uh, characteristic to it. If it's a fingerprint, if it's a specific, uh, public key. If it's, uh, that there's a number of, small number of, uh, Expected good fields in the X 5 0 9 3 that you can use for 

[00:08:01] Matthew Coles: generally immutable, generally immutable fields that that

[00:08:03] you wouldn't expect to be able to be tampered with easily.

[00:08:07] Izar Tarandach: And then basically you're saying after we did all the crypto validation, after we did all the chain of trust validation, I'm going to look at this one thing to see if it's the specific certificate that I'm looking for. And for example, one of the things that this breaks is, again, if you are in an enterprise, uh, environment and they have, uh, uh, uh, interception of encrypted, uh, content for D lp, for whatever reason it is, that's when you're going to see this thing break.

[00:08:39] Chris Romeo: So if I use, let's encrypt, which, which, uh, has a 90 day life cycle by default on certificates. Do I have to refresh my certificate pin information every 90 days when it refreshes and gives me a new cert for my application?

[00:08:55] Izar Tarandach: the best of my, my, uh, understanding, and I could be wrong here. You do have the, uh, you can't play games with the valid. Not valid before field and things like that so that even if the, the certificate is expired, you, you can still do the validation, but somebody who's, who wants to be actually strict about that, that kind of thing.

[00:09:19] If, for example, they are writing a mobile, uh, uh, application, they will have to reissue the, the whole app when the the, when the certificate that they're pinning changes.

[00:09:32] Chris Romeo: To have

[00:09:32] the new value.

[00:09:33] Matthew Coles: Alter Alternatively, uh, what you, what you pinned to is the parents or the issuer such, such that, such that when I doer validation, if the issuer of the, of the presented server cert is not the pinned certificate. I'm going to ignore it cause it's not good, right. From my perspective. So keep in mind a server can present whatever certificate it wants, the browser's going to accept whatever is in the cert store. Um, and, uh, or, or, you know, validated through the cert store. And, and in theory, that certificate is associ. The leaf cert or the, the presented cert is, um, is associated with the host that you're talking to. And you know, if you do o and OU validation, right? But it may be issued by somebody else.

[00:10:19] Than you expect, right?

[00:10:20] So the pinning is really, is really not, did I get a cert that has a trust chain? That is correct. It's did I get a certificate that meets my criteria for the one I want?

[00:10:33] Specifically? 

[00:10:35] Chris Romeo: All right, so now I'm gonna ask a question that, that you guys are just gonna hate having to answer, but that's okay. I'm gonna do it anyway.

[00:10:42] Matthew Coles: What else

[00:10:43] Chris Romeo: are the. What are the, what are the, what are the threats? This is a complete joke cuz you're gonna love this question. What are the threats that I'm trying to counter here? Like if I do, let's, let's threat model, like let's threat model a a system or an application, let's say that doesn't have certificate pinning. What are the threats where, where threat or certificate pinning is the mitigation to counter the threat?

[00:11:08] Matthew Coles: So the, so one of them, uh, an easy one here. Just I, I'll bring it back to the, did I get the one I expect the thread is from version through another trust chain. So, Right. So if a certificate could be issued for same host against two parents, then, uh, or, or by two parents, uh, and you wanna know you're connecting to one versus the other, um, that's, that's the, the thread here is that somebody could, somebody who owns a root A, the ability to sign from a root certificate, but is not the one you expect. Could create a legitimate cert that you would then attempt to communicate or, or talk to that server with, but you don't trust that chain, even though the certificate chain is trustworthy in terms of its validity and its signature and its et cetera, it hasn't been revoked, et cetera, but it's not the one you expect. So you could

[00:12:02] be 

[00:12:02] Chris Romeo: is that.

[00:12:02] Matthew Coles: spoof service or to

[00:12:04] Chris Romeo: Is that a malicious, so, so you're saying that that secondary chain that I, I'm, I'm interacting with the web app, I'm trusting it and I shouldn't, that would be potentially a malicious

[00:12:14] Matthew Coles: Potentially,

[00:12:15] Chris Romeo: who created that, that secondary

[00:12:17] Matthew Coles: potentially malicious. Although I think in practice, what I've seen, uh, is really, um, not malicious, but, but probably not the one you, you want in terms of service capability or like an iot device, for instance. Right. Talking to a. Um, has iot devices have limited capabilities as it is, uh, in some cases, right?

[00:12:38] They don't necessarily have the ability to do full cert validation in some cases, right? Because of algorithm choices and, and software versus harbor implementations. Um, and so your, your threat isn't necessarily always going to be that the server is malicious, but, but maybe it just isn't the one you intend to be communicating with.

[00:12:58] Chris Romeo: Okay.

[00:12:59] Izar Tarandach: So at, at the end of the day, it stretched against the spoofing of the other side and tampering with communication in the middle. And, uh, we, we have had enough, uh, examples of. Not on application side, but on the issuer of certificates doing some crazy stuff and issuing a perfectly valid certificate to someone who shouldn't have it with good names and all that.

[00:13:25] And, uh, the fact that you, you can be more selective in terms of which specific certificate you're going to accept rather than just a generic certificate that is valid based on its chain of trust. It just gives you that one more selection towards, uh, um, again, spoofing. But, uh, uh, at, at a, at a, at a business level thing, I think it makes you doubly sure that you are talking to the provider that you are talking to and not to someone who just ended up having a certificate that's, uh, uh, sufficiently looking like the one that that person would have.

[00:14:09] Matthew Coles: Right. And, and again, I guess the other piece here is the other, I'll just mention and I, I don't know if there's a threat, it's an injection threat, I guess, at the bare minimum. But, um, as I mentioned, you know, some, some devices may not have crypto cryptographic capabilities. To perform

[00:14:24] proper validation of a either of a chain of a certificate and its issuer, meaning its signature and, and that, so there you're making an explicit choice to trust a particular cert or set of certs, um, in lieu of the validation that you would do during traditional X 5 0 9 P k I, uh,

[00:14:43] uh, exchange.

[00:14:45] Izar Tarandach: Which it's not ideal, but for that class of device, many times it's the only thing that can be done. 

[00:14:51] Matthew Coles: That's right. 

[00:14:52] Izar Tarandach: better than nothing.

[00:14:53] Chris Romeo: Okay.

[00:14:54] Matthew Coles: Better than, better than not having tls. 

[00:14:57] Chris Romeo: yeah, definitely, definitely. All right. So I feel like you guys have really, uh, you guys have really educated me here today around the security table. Like, I, I learned something and, you know, hey, I think there, there's a lesson here for people out there and, and it's taken me 20 plus years of my career. Guess what? You don't know everything. I don't know everything. I'm not afraid to ask people when I don't know something at this point in my career because I, it helped me. I now have an understanding of certificate pinning that I'll carry with me for at least six months before I forget about it and have to ask these guys again, 

[00:15:26] Matthew Coles: And by the way, you know, we, we could be

[00:15:28] Izar Tarandach: Totally. Yeah. So check 

[00:15:30] Matthew Coles: uh, get, get, get comments here. Definitely feel free to, to counter counter us. And this is by the way, I think not traditional, not expected browser behavior, right? Uh, the folks who set the rules for browser behavior, this would be very bad, I believe.

[00:15:42] So this is definitely a non-standard thing, uh, for a lot of people. But it does come out, uh, come up in situations often. And in fact, I talk about certificate pending frequently, um, and not just in the TLS case. Right. Uh, there's other uses of certificates, obviously, where Sir Painting has validity as a mechanism, but something you've touched upon.

[00:16:02] Chris, I know we, we sort of had a, an agenda for today, but I, you know, I think it's really important what you just highlighted, um, about. There is so much stuff for us to know as security, as security professionals. We're not even developers. We're just, we're security people. Right. And, you know, and it, it's just, and I think about this actually pretty free frequently, you know, I'm, I'm. Obviously I'm not, I'm not a spring chicken anymore, and, uh, and I'm, I've got a lot of years under in my career, and so a lot of people come to me for advice and guidance and I work with a lot of teams, you know, in, in my day job and, and. The breadth of technology and the business domain and the regulatory domain, and I'm, and that's just security, not to mention privacy and and other as safety and, and consumer use and a whole host of other things. It's like, it's like there's a, there's a bucket of information and then at some point it starts flowing off the top and you don't know what you're losing off the top right. Um, I always joke that I, I joke sort of halfheartedly. I do have a sticky note here somewhere that I have a conference presentation waiting to be written called t i l, right?

[00:17:15] Today I 

[00:17:15] Izar Tarandach: Nice. 

[00:17:16] Matthew Coles: probably something, either something I forgot or something I didn't know that I needed to know, that I didn't have the skills for originally. We're in a constant learning pattern. People talk about ai. I'm like, I, I don't know. Ai, I don't even have the capacity for it. Kubernetes, sometimes that's a stretch, although, uh, you know, that's happening more and more frequently for me at least.

[00:17:36] Right. Others are probably already steeped in that, whereas device security Yeah, that's, I, I can, I can tackle that. That's in my, still in my bucket of, of knowledge. Right. So it's really important. I think that, and I don't know what you're, you know, actually I do have a question for you because actually I thought about this. How do other, not only highly successful people, but other security practitioners especially, um, how do they deal with this? Like everyone's, a lot of, a lot of people seem to be very far ahead in, in a lot of areas and like keep a lot of stuff in their head and they know about it and they, and like, like the trivia game, like, I wouldn't be good at Jeopardy, but some people are really gonna be good at security jeopardy.

[00:18:16] Right. And, and as there's esoteric knowledge and there's really some basic stuff. And so how does, how do other people make this work?

[00:18:23] Izar Tarandach: So I, I don't know about highly successful people, but I can tell about myself.

[00:18:30] just say that every time that a new release of Chrome comes out and I go through the release log, the only thing I'm looking for is stuff that's connected to tab management, cuz anything that can help me manage 200 tabs that I have open at any given time. That's the valuable stuff for me. So yeah,

[00:18:52] Matthew Coles: Have you discovered the reading? Have you discovered the reading tab, the reading Pain? By any

[00:18:56] Izar Tarandach: I, I, I did, I think that I, I must have found a buffer overflow in there. Cause I just put one more and the whole thing closed. But, uh, , you know, I, I, I, I think that, uh, one of the most powerful tools that, uh, that I found in, in that sense. And, and Matt, thank you so much for bringing this up. Like this is something that I, I don't think that we can explore enough.

[00:19:20] I think that one of the most powerful tools that I found to deal with this was the ability to get myself to sincerely tell someone I don't know. I, I'll get back to you of an answer, but right now, at this moment, I don't know, and I don't want to talk out of my behind, so I, I prefer to shut up and come back to it later than to start moving my hands very fast in the air as I usually do, and, uh, and say something that I'm going to regret later.

[00:19:51] Chris Romeo: I I mean, that's wisdom that what you're, what you're describing there is wisdom though, because I would've, I almost the same way early in my career, I felt a pressure to answer. Oh, I'm on the spot. I gotta, I gotta have an answer. So I gotta dance around and try to, try to dance in such a way that the answer makes sense and people don't, don't think that I have no idea what I'm talking about, but the older I get, the longer I've been in in this industry, the more I'm like, that's, I have no idea what you're talking about. Like, I probably have an opinion if you, if I, let me ask you a couple of questions to you so you can teach me and explain it to me and, uh, you know, I'm an expert in my own opinion. That's, that's something I'm an expert I'll claim expertise in. But, so I think we've, for the folks like us that have been around for a long time, we've, we've seen so many different things that we can, usually, we can form an opinion about the security of something, but often we have to ask questions like, okay. Tell me more about this. How does this work? What is that thing you just, you just said that word. I don't know what that means. And once you tell me about it, then I can give you, at least give you an opinion on it. But I, I think it's important. I think that's something everybody can learn though, is nobody has all the answers in this industry.

[00:20:55] And if somebody claims to, they're run, run away from them quickly because

[00:20:59] Izar Tarandach: And, and to take this to the place where we double quote, leave. When you are threat modeling and somebody comes to you if a new design, and sometimes you'll say, Hey, in here we put an AWS Fox Digger SuperDuper new box. And you know what? I have no clue what that service does. I have no clue how it's configured.

[00:21:21] I have no clue how it can fail. I, I have to tell people, Hey, wait, I'll, I'll go check this and get back to you, but I'm not going to start, oh, csrf, rf, whatever, f Right. And unfortunately, sometimes that happens. And, uh, we have to be, to be honest enough to say if I do that, I am actually putting a, a. Uh, putting a, a bump in on the road, which I don't have the right to do.

[00:21:47] I'm here to facilitate things, not to make them harder. So, yeah, we have to leave our egos in the, in the door when we do that.

[00:21:56] Chris Romeo: That's good advice. That's good advice. Well, I tell you what, we're gonna, we're gonna end this episode of the security table, uh, where we, we kind of started diving into certificate pinning almost on accident. I learned what it was, but then we had some reflections on our careers in the industry and some advice for people that are new to it.

[00:22:13] But, uh, stay tuned. We'll be back shortly with another episode of the security table.

[00:22:21] ​