March 4, 2021


These days, everything is connected to the internet. Whether it’s your car, your light bulbs, your microwave, your pacemaker, or your cochlear implant, it’s all being run and dictated by the internet. 

And with that brings a whole new set of concerns. 

Where you used to just have to worry about keeping your bank account secure, or your home wifi network secure, now all of a sudden you have to worry about your car or your pacemaker being hacked?

How do we even go about categorizing all the IoT devices, and how do we protect them?

On this episode of Virtual CISO, I chat with Aaron Guzman, who in addition to being the Product Security Lead at Cisco Meraki, is also the Project Lead for the IOT Security Verification Standard (ISVS) at the OWASP Foundation. And if that wasn’t enough, he’s the author of a number of books on IoT, including  IoT Penetration Testing Cookbook. 

He was kind enough to talk about:

  • OWASP
  • What the ISVS is
  • Who ISVS is intended for
  • And, how ISVS is categorized 

To hear this episode, and many more like it, you can subscribe to The Virtual CISO Podcast here.

If you don’t use Apple Podcasts, you can find all our episodes here.

Time-Stamped Transcript
This transcript was generated primarily by an automated voice recognition tool. Although the accuracy of the tool is 99% effective you may find some small discrepancies between the written content and the native audio file.

Narrator: (00:06)
You’re listening to The Virtual CISO Podcast, a frank discussion providing the best information security advice and insights for security, IT, and business leaders. If you’re looking for no BS answers to your biggest security questions or simply want to stay informed and proactive, welcome to the show.

John Verry: (00:25)
Mr. Guzman, good afternoon, sir. Or actually, probably good morning still for you or just noon for you, right?

Aaron Guzman: (00:31)
It’s just noon, yep, yep. Happy Friday.

John Verry: (00:34)
Thank you. I’m on the East Coast and Aaron’s on the West Coast, so appreciate you coming on today. I think you may be… I was trying to think about this beforehand. We’ve been doing this almost a year. You may be the first two-time guest, so condolences on that.

Aaron Guzman: (00:49)
Awesome. Thanks for having me.

John Verry: (00:51)
Well, you did such a good time the first time, and the topic that we’re here to talk about today, the IoT and security and testing, I think it’s such an interesting and intriguing and fun topic. You wear a bunch of different hats in this space, and we’ll get to that as we go along. Let’s keep it simple to start. Tell us a little bit about who you are and what is it that you do every day?

Aaron Guzman: (01:13)
Yeah. Aaron Guzman. Again, based out here in LA. I lead product security for Cisco Meraki. And essentially, our team is responsible for basically having the customer’s back and building security features on their behalf and ensuring our vulnerabilities and our vulnerability management program is up to speed and ensuring that we’re meeting those SLAs to ensure, again, customers aren’t vulnerable, things like that.

Aaron Guzman: (01:42)
But outside of work and my day to day and my team, I lead a number of OWASP projects around IoT and embedded application security. Then I’m the co-chair of the IoT Working Group with Cloud Security Alliance, which was the last podcast I was in right during the beginning of football season, I remember, when we were going over that.

John Verry: (02:03)
Didn’t know if we’d have a season.

Aaron Guzman: (02:04)
Yeah. Right, right?

John Verry: (02:06)
And it happened. It happened, amazingly.

Aaron Guzman: (02:08)
Yeah, yeah. So pretty involved in the IoT space. Even outside the two groups that I help lead, collaborate with other groups, I and the cavalry, and cross-collaborate with ENISA and some other European-based IoT groups, again, to make sure that we’re all saying the same thing in different ways, but trying to share that same message. Everyone’s doing the mapping, but having fun. I love this stuff. I wouldn’t spend as much time and devoting extra time and helping mentor folks-

John Verry: (02:45)
Endless waking hours would be how I would define it, but that’s the thing.

Aaron Guzman: (02:48)
Yeah. They’re all separate streams of efforts, too. It’s not just me. It’s definitely the project teams, the contributors, the volunteers. They definitely help out a lot, especially when you’re bogged down at work and you just can’t get the extra time and the mental capacity to get that energy. So this is a brief intro.

Aaron Guzman: (03:12)
I’ve also wrote a book in 2017 on IoT pen testing. It’s called the IoT Penetration Testing Cookbook. And I’ve edited a few IoT-related books as well, Practical IoT Security with Brian Russell and Drew, who Brian Russell helps lead the CSA IoT Working Group with us. So we do that together. I recently finished up, it’s called Practical IoT Hacking with No Starch Press, which should be released in a couple of weeks, the digital versions.

John Verry: (03:42)
I should’ve asked you. I haven’t seen that one, so that’s cool.

Aaron Guzman: (03:45)
Yeah. The digital version’s out. Super comprehensive book, very technical. Coasts the whole gamut of IoT from hardware to wireless to RFID tags to ISE Core C, lower level hardware debugging and sniffing. I’d recommend-

John Verry: (04:04)
You have to send me a link to that. You have to send me a link to that because by the time we publish this, that’s probably going to be out. So definitely, we’ll promote that. We’ll also throw the Penetration Testing Cookbook, I think you called it, right?

Aaron Guzman: (04:15)
Yep.

John Verry: (04:16)
I’m familiar with that book. I think we own a copy of it, actually.

Aaron Guzman: (04:19)
Awesome.

John Verry: (04:20)
I guess you’re one of those people that is of the I’ll sleep when I die camp.

Aaron Guzman: (04:26)
Yeah.

John Verry: (04:27)
You make me feel very inferior. I mean, you ask me what I do, I do this and that’s it. That’s all I do, and you just never stop. So congrats. I appreciate it and definitely appreciate all your contributions to the industry. Because you do, you put in a lot of your time, I think, to make the world a better place. I know that sounds a little whatever, flowery, but it’s the truth. And I appreciate all the folks at OWASP. I’m an OWASP fanboy. We’ll get to that in second. Before we get there, though, I always like to ask folks, what’s your drink of choice?

Aaron Guzman: (05:00)
Last time around I was on a health kick, and I said one of the diet drinks. Lately, I’ve been on a beer kick, and the hazy IPAs are what I’ve been favoring or even Lagunitas is something that-

John Verry: (05:12)
A little something-something.

Aaron Guzman: (05:13)
Yeah. Yeah. A little something-something, the 12 percenters.

John Verry: (05:17)
Yeah. Oh, yeah.

Aaron Guzman: (05:17)
I’m familiar with those.

John Verry: (05:18)
Well, you also have the advantage, I mean, you’re on the beach. You live out in LA, so you get a chance to spend more time… I’m looking out the window, it’s snow. So a beer of that nature is really good. I mean, have you ever had any of the Founders, what they call an All Day IPA? Have you ever had that?

Aaron Guzman: (05:37)
All Day IPA, yep. Yep, yep.

John Verry: (05:38)
Yeah, which is great. Because I mean, a good IPA is wonderful, but you get to two or three and you’re like, “All right. I’ve had enough.” But the All Day IPAs are nice because if you’re out over an extended period of time, you can drink a few of them and it doesn’t really catch up with you.

Aaron Guzman: (05:54)
Yeah. Hopefully, right. Yeah. IPAs, depending on what they are, alcohol percentage, those sneak up on you.

John Verry: (05:59)
Absolutely. You always got to look. Somebody hands you one, you got to look.

Aaron Guzman: (06:03)
Yeah.

John Verry: (06:03)
All right. I’m an unabashed OWASP fanboy, and I’ve chatted with a lot of folks from the OWASP organization. But briefly, just tell us who is… Because that’s the subject of today’s conversation, who is OWASP and what do they do in general?

Aaron Guzman: (06:18)
Yeah. OWASP, it stands for Open Web Application Security Project, and they started in the early 2000s. It’s a not-for-profit organization that’s global. They have chapters regionally around the world. The goal of OWASP is essentially to put out software security guidance, tools, resources. With the chapters regionally, they have community networking events. Right now it’s a little bit challenging, although they’re starting to revamp the membership as well. Membership’s, I don’t know, $50 a year. It’s not really much, and they’ve started providing these trainings and educational courses just recently.

Aaron Guzman: (06:59)
So there’s some good stuff coming from OWASP, and I mean, I can give a lot of credit to OWASP for where I’m at. I’ve gotten so many opportunities and jobs from attending OWASP meetings here in Los Angeles. I was on the board for five years, going to the global conferences as well. The content that we put out is pretty authoritative in software security, and that’s one of the reasons why we talk about ISVS and we’ll talk about why we did it a little bit more, but it’s a great platform as well.

Aaron Guzman: (07:35)
People are familiar with the name if you’re in application security or anything security, really. Or if you’re a developer who has been handed a task to be able to remediate or fix a vulnerability, you’ve likely looked up OWASP in some way, shape or form.

John Verry: (07:50)
Yep, yeah. Listen, I look at OWASP as being the foremost authority on all things application security. What I love about it, OpenTrust is standard, largely volunteer led. Most of the folks are like you. They’re people that have day jobs, and they’re doing this because they love it and because they’re trying to make the world a better place, right? Software runs the world, and if software’s insecure, we’re all insecure. That’s OWASP, and it’s a fantastic organization.

John Verry: (08:17)
You chair a couple of projects. The one we’re chatting about today is the ISVS, right? The IoT Security Verification Standard. Tell us, who is that intended for?

Aaron Guzman: (08:28)
Yeah. Well, first, I’ll get into who it’s intended for, but essentially what we wanted to do with the project is because ASVS reached out to us, I think, a couple of years ago with their last release for an IoT section.

John Verry: (08:48)
Real quick, real quick, Aaron, the ASVS is the Application Security Verification Standard, one of the hallmark, one of the best documents in the world, I think, with regards to securing applications. So you wrote an appendix that I think is where you’re going to go. I just wanted to make sure if someone’s listening, they caught up with what that meant.

Aaron Guzman: (09:05)
Exactly, yep. OWASP has a few different verification standards, and ASVS is one of the flagship. Then they have also the Mobile ASVS, and now they have a software composition verification standard, SCVS. I forget-

John Verry: (09:20)
The other one is for S bombs?

Aaron Guzman: (09:22)
Yes, yes. That’s one of the functions, for sure. One of the goals of our project is to be one of those kind of references that is similar to the uptake of ASVS and be that kind of open standard that folks follow for a number of reasons and a number of uses, like being able to have a secure or security by design, however you want to put it, kind of framework from the get-go when you’re designing products. And not only when you’re designing, but also once a product is already in the field. It’s really intended for everyone from product managers to product security teams to assessors, like penetration testers, or even security architects. It’s a wide range.

Aaron Guzman: (10:09)
One of the main goals when we were developing the standard, and by the way, probably by the time this airs, we’ll release it. It’s supposed to be the first week of March is the release date, the first Friday, so I think that’s around the 5th. But when building this standard, we wanted to make sure everything was actionable, measurable, testable. Minimal ambiguity, really, because that’s where all the other IoT security guidance and standards are. It’s very high level, and it’s easy to just say, “Oh yes, we do that.” Or maybe it’s too long and there’s a number of different requirements or testable kind of tasks in there, but it’s not really straight to the point.

John Verry: (10:57)
I mean, yeah. Like the NIST guidance, right? I mean NIST puts out great guidance, and I love NIST, right? They put out great… But specifically in this area of 8228 and 8259, they’re not quite where they need to be for somebody like us to look at this and say, okay, how do we test this, right? What is the process by which we test this? I think it would be the same thing for someone, how do I design and build something that’s going to be secure when it gets out into the field?

Aaron Guzman: (11:20)
Exactly. I actually reached out to them for some input and feedback and shared with them the ISVS. Going to be scheduling a call to get going on helping out because I feel like NIST is doing some great things. I’m definitely more involved… It’s really odd that I’m more involved in European-based… You know?

John Verry: (11:42)
I think NIST’s guidance on IoT stuff is fantastic. I mean, I think they’re really ahead of the game. I mean, I like the fact that they’ve segmented IoT into different worlds, different fields of use, right? Smart transport, smart cities, smart vehicles, right?

Aaron Guzman: (11:56)
Vehicles, yeah.

John Verry: (11:57)
And general IoT. So I think they recognize… I mean, I think they’re ahead of the curve in terms of their thought process on IoT.

Aaron Guzman: (12:04)
Yeah. They definitely make use of resources, stakeholders in the industry. Yeah, definitely have open communication with them. We often use the baseline that they have. Even with the CSA IoT Controls Framework, we use them both and we reference them both. If you look in ISVS, we make sure to add them as a reference point of people to be aware.

Aaron Guzman: (12:28)
But coming back to the ISVS and some of the other things to note is since I mentioned ASVS for general web application and the mobile, we try to ensure that we, where possible, we reference other standards. Let’s say if it’s not only for web and mobile but even for cryptography, NIST has some great guidance and things that we shouldn’t re-create the wheel there more than just highlight the most important areas, which we have, and then reference where it makes sense.

John Verry: (13:06)
You have a graphic in there. I haven’t done this before, but I’m going to try to do this and open up this, share this. We’ll see what happens. If anyone’s watching on our video feed versus the audio feed, hopefully you’re seeing what I’m showing. But I really like this because when I first opened up the ISVS, I was trying to, as anyone does, find their way in and what’s the thought process here. I, of course, jumped to the table of contents, and I was figuring it out. I slid through here and I went, “Oh, this maps perfectly to the table of contents.” And I thought this was a really cool way to look at this, right?

John Verry: (13:42)
We’ve got these five, I don’t know what you’d call them, sections, areas of concentration. Then within them, we have these 18 specific areas of concentration. Then within that, you’ve got, if I recall correctly, what, 124 different requirements across those 18 different areas. Could you walk through? I thought this was just an elegant way to help people understand the complexity of securing an IoT ecosystem?

Aaron Guzman: (14:10)
Yes, yes. Absolutely. The way you would look at this image is bottom up and how even from when a product comes into what they call NPI, new product introduction, you start with product requirement documents, PRDs, right? The PRDs where it defines all the fun stuff that that device is going to do or that product. For us, we start reviewing the hardware PRDs first. And then there’s also software feature PRDs, and that’s where you input some of the more drill-down details of what the product is going to be or should be following from a requirement standpoint.

Aaron Guzman: (14:49)
That’s where the software platform to the left, which is the bootloader, everything after once the secure boot chain has finished, everything to the user space applications on top, software updates. And then we have kernel space as well. That’s the way usually product teams are structured and who’s responsible for that particular area are platform teams. Wanted to make sure that we communicate how teams actually work.

Aaron Guzman: (15:24)
Then I, obviously, have that insider knowledge. I’m fortunate to have that experience and worked in different product companies. So being able to relay that to the proper team from the software platform and then going over to the communication and user space applications, that’s usually the higher level product teams who are working on different types of features. There’s different subcategories in here within communication, user space applications.

Aaron Guzman: (15:53)
Of course, there’s general things that apply to any type of application or environment or device. We wanted to make sure that, again, we covered what is particular and what is the most common kind of use case for IoT and embedded devices. The kind of approach with some of these is there are some device or product level requirements, and then there’s ecosystem level requirements. As you see at the top there with application ecosystem design, secure development, supply chain, and supply chain will feed off of what SCVS has done. Secure development, we feed off of ASVS. Even in our control objectives at the top of each section, we make sure to know and reference our sister projects, right, because that’s what they are.

Aaron Guzman: (16:45)
It’s pretty drilled down as far as from a device perspective, but also the ecosystem, right? Because IoT is usually a systems within systems, so it can get as complex as you want.

John Verry: (16:57)
There’s a level of inception to IoT testing, I always tell people.

Aaron Guzman: (17:01)
Exactly. Everyone has their perspective on the category of IoT. Connected vehicles, for example, they have their own complexities in the way they interact with each other. We try to hit the breadth of all IoT from that perspective, but there are particular things, like I gave automotive as an example, where it’s a little bit difficult to give that specific kind of guidance and best practices where it wouldn’t really apply as much to the rest of the kind of category of IoT. Could be too niche.

Aaron Guzman: (17:37)
As an example, we have some folks that, say, ask us to put in some log integrity and some kind of forensic-based controls. I think that’s a little bit out of scope, right, because part of the reason why OWASP is the platform here that we’re using is everything is software. From hardware, from wireless communication, Bluetooth, everything’s a library. Everything’s a file. All the chips, everything is software, and it’s going to continue to be software.

John Verry: (18:10)
It’s only getting worse, right.

Aaron Guzman: (18:11)
Yep, yep.

John Verry: (18:12)
Software’s eating the world, right?

Aaron Guzman: (18:13)
Exactly. Yep, yep. I mean vehicles have like 300 million.

John Verry: (18:18)
Oh, yeah. I mean you think about it, you’ve probably got more lines of code in a Tesla riding around than you did in the original version of Windows, right, Windows 3.1. right?

Aaron Guzman: (18:27)
Absolutely, absolutely.

John Verry: (18:29)
Which is a nutty thought. One of the things that I also like about what you guys did and it was something that I liked that you did with the CSA IoT Framework as well is you scaled this to take a risk-based approach. So we do have three levels like we do with ASVS and the other frameworks that adopt a similar… Talk about level one, level two and level three and a little bit about what the intent is for each of those levels and maybe an example or two of where someone would pick one versus another.

Aaron Guzman: (18:59)
Yeah. At first, we were approaching the levels like difficulty to attack. We started with level one is software only protections against attacks, don’t really require physical access to it, to a device. It’s really like the baseline, level one. Level one should be the baseline of where you start. We have noticed some examples within the project page just to give some context, right? Like smart cameras, I’m sorry, that’s level two. But light bulbs, some simple devices that, let’s say, may have minimal feature sets. Or you could have some that are just basic consumer off-the-shelf products, right, where there’s a base level of requirements that, let’s say, a partner, a third party defines for them and that’s all they want to adhere to.

Aaron Guzman: (19:59)
But level two provides protection against both software and hardware attacks. Examples are alarm systems, smart cameras, medical devices for level two. While level three is to provide requirements for devices that should be avoided at all costs, and we think of these as hardware crypto wallets, vehicles, medical implants, and so forth. It’s really the highest level of security. Maybe it has safety concerns, direct safety concerns that could affect you, or monetary concerns as well.

Aaron Guzman: (20:36)
It should provide a high level of assurance and trustworthiness, meaning the device or product or a vehicle should always function as it’s designed. Even if it’s being attacked or even if it’s trying to be modified, there are controls in place to hinder those types of attacks. Reverse engineering and tampering is some of them, although we didn’t go too far into that rabbit hole. Because you can really go deep as far as the layers you put onto resiliency, although it’s a topic that we’re considering for a future version similar-

John Verry: (21:18)
Yeah. MASVS, we’ve just recently been doing a lot more level one plus R. And man, those requirements are very difficult. I can’t tell you how poorly most clients are doing with R, so I think the R is a really good idea because I don’t think people… I think when you look at design teams, I don’t think that they’re designing for resilience.

Aaron Guzman: (21:41)
Yeah. It depends, right, on the industry. You really have to be doing the basics, the fundamentals first, but definitely keep the resiliency part, the tamper protections in mind. But if the basics aren’t done, all the work you’ve put in for the resiliency areas, attestation and things like that, and it’s also product dependent and even chip dependent.

Aaron Guzman: (22:07)
That’s the complexity with embedded devices. You’re usually reliant on a partner of some sort, and that could be a semiconductor partner. That could be original design manufacturer or a contract manufacturer, just a supply chain and how things are sourced. You’re sometimes constrained by those choices that you may not even have an opportunity to give feedback, and maybe security teams aren’t even looped into those discussions.

John Verry: (22:37)
Yeah. And that same set of challenges on working with partners or vendors or suppliers that you’re dealing with on the tech and on the hardware side, you have the same set of challenges on the software side, right? I mean you’re going to run an RTOS or you’re going to run some version of an embedded Linux. I mean you’re using people’s libraries. I mean, you got the same challenges.

John Verry: (22:56)
Quick question for you, so when I was looking at your level one, level two, level three, in my brain I was perhaps oversimplifying it. But level one is someone can get to it from a connectivity perspective, right?

Aaron Guzman: (23:09)
Yep.

John Verry: (23:09)
It’s visible to somebody. Level two is somebody can get the device in their hands or get to the device. Level three is somebody is willing to take apart, manipulate the device.

Aaron Guzman: (23:22)
Exactly, yeah. It requires, let’s say, expensive tooling, maybe even a lab, things like that.

John Verry: (23:28)
Yeah. Yeah. Actually, we’re right now having conversations about converting our testing model to follow this. But actually, you followed our testing model, right? We start with looking at it from the network and from an application visibility, and then we drill down to what can we do without messing with it and what can we do when we do. When we get to level three, you’re opening things up. You’ve got your solder guns. You’re looking for JTAG, SWD, UART, some type of interface into it. Can we take activity, can we interrupt it and screw around with the bootloader?

Aaron Guzman: (24:08)
Yeah. Side-channel thing glitching.

John Verry: (24:09)
Exactly.

Aaron Guzman: (24:10)
Those are difficult, you know? Difficult to prevent from. You know what I mean? You have to work with the partner.

John Verry: (24:15)
Oh my God, yeah. Well, and in fairness to the people that are doing this, you just wouldn’t think that somebody’s going to do… It’s not even a risk that you’re thinking about as a developer. I mean, I feel bad in a way that there’s a group of devices we just beat up pretty badly, and they left holes in the case to get to the debug SWD, right? Because from their perspective, it was like, “Oh okay, that way we can troubleshoot something if we have to.” But they didn’t fully understand the implication of making those holes in the case, so we didn’t even have to open up the case to do what we needed to do.

Aaron Guzman: (24:49)
Yep, yep. That’s-

John Verry: (24:49)
You did some really cool stuff in there. It’s so funny because I’m reading your document and I’m QA-ing somebody else’s document on the testing. Literally, I’m reading one line in your thing where it’s like, you should remove silk screening on the board, which a lot of people would think is a crazy idea, right?

Aaron Guzman: (25:07)
Yeah.

John Verry: (25:07)
But yet I’m reading our pen testing narrative, and they’re saying, we’re trying to determine which port would be… But this one was labeled as debug. It was labeled as UART or JTAG, so we knew that, so it helped us figure out. We literally made the recommendation before your document was out-

Aaron Guzman: (25:26)
Nice.

John Verry: (25:26)
… maybe you should remove the silk screening. So it was like, “Wow, that’s kind of a cool connection we’re seeing there.” You know what I mean?

Aaron Guzman: (25:31)
That’s a controversial one. I believe in it. I believe in layers.

John Verry: (25:34)
I think you’re right. I think you’re absolutely right. It’s a form of obfuscation, right?

Aaron Guzman: (25:38)
Yeah.

John Verry: (25:38)
I mean at minimal you’re going to slow somebody… I will tell you that-

Aaron Guzman: (25:41)
Exactly.

John Verry: (25:43)
… when I read our pen testing narratives and we don’t know what this block of pins does, we’re less likely to put as much level of effort into breaking it, right, because we’re not sure if we have a high probability of success. But if it’s labeled that way, it’s like, “Yeah, we should be able to figure this out.”

Aaron Guzman: (25:59)
It takes a lot more experience in the area, a lot more tooling. You’ll need to be able to identify different pinsets, if it’s 10, 14, four. Is it UART? Is it SWD? Is it whatever it is, ISE Core C? Yeah, it’s just one obstacle. It raises that barrier of entry, right? Because you have oscilloscopes and logic analyzers and test points and things, things that complicate it. You have to know how to read those. You have to know what protocols and how to read the protocols and demodulate those, whatever it is.

John Verry: (26:33)
Yeah. I will tell you this, is that the level of learning that we’ve had to do over the last couple of years and the amount of money that we’ve had to spend on tooling devices, and you’ve seen the industry grow up too in that now instead of having to actually build your own, now you can jump online and buy… JTAGulator is an example. Right? Which again, to your point, is a valid reason for not having silk screening on a board because the minute we get to a point where a script kiddie can buy a tool and maybe attack a device, we want to do everything we can to prevent that from happening, right?

Aaron Guzman: (27:10)
Yep. And that’s only one way. Because there’s secure JTAG, but there’s no standard for secure JTAG. You’re pretty much dependent on the chip vendor or whatever trust anchor you’re using to be able to provide that protection and monitoring the JTAG bus and pretty much killing the JTAG port, essentially, when it recognizes current or electrical pulses, if you will. But yeah, I mean those are some of the things we’re also thinking about, but I know in practice it’s not easy.

Aaron Guzman: (27:40)
So we strayed away from… If it’s not practical, we’re not going to put it in here. If it takes an extreme amount of design time, the teams will know about these certain things, like attestation is one of them. Things are cloud based now. Attestation could be for a traditional on-prem device, right? And I could think for the mothership of Cisco that I work for, that’s something that they advocate for, but with Meraki, it’s a different business model, I guess you can put it.

John Verry: (28:11)
There was a line in the document that I liked. The ISVS therefore specifies secure requirements for embedded applications in the IoT ecosystem in which they reside while referring to existing industry-accepted standards as much as possible. There’s two things I really loved in that sentence. First was the use of the word ecosystem.

John Verry: (28:30)
So I’ll ask you first there, there are many device guidance, IoT device testing guidance. Why the choice of the word ecosystem? How applicable do you see ISVS as being to the ecosystem, and how broadly do you define that term ecosystem, right? Because in IoT, that ecosystem can extend literally to dozens of organizations, potentially, depending upon the type of device.

Aaron Guzman: (29:00)
Yeah. That’s a great question and something that we had been thinking about. Because when you look at some of the guidance in the section, you’ll see machine to machine, and then you’ll see the wireless protocols, right? That’s also a form of depending on how the implementation is, could be a machine to machine. But we want to keep it simple, for sure, as concise as we can and clear, if that makes sense. There is a focus on device or product level down to the firmware things, and then like I said, we piggyback on ASVS and MASVS where we can and NIST for Bluetooth and the cryptography.

Aaron Guzman: (29:45)
I think what we’ll need to do at some point is be able to provide some more visuals, some more graphics that can help with where we draw the line and the boundary. We do have a basic one up, and that’s something to just give you a better idea of what the scope that we’re talking about but more of how it looks in a real environment, I guess you can say. Not a real environment, but from a data flow diagram, if you will. This is something that we did with the Cloud Security Alliance IoT Controls Framework to be able to help with that context again. But yeah, I definitely-

John Verry: (30:19)
I was going to say, so from my perspective, when I look at this visual of what it might look like, so we’ve got the device in and of itself. That device is talking to other devices in the local environment. Those devices may or may not serve local APIs, local web services. I might have a mobile application that’s being used to configure or operate said devices. That mobile device might be consuming APIs or services off of the local device, might be talking to the cloud.

Aaron Guzman: (30:46)
Exactly.

John Verry: (30:46)
The devices themselves might be talking to the cloud. I may have a go-between. Then you’ve got the web app with the cloud, the APIs. Then, even then, those APIs may be being exposed to partner ecosystems. If it’s a smart speaker, you’ve got Spotify and we’ve got Pandora and people of that nature. What would be cool to see is OWASP’s vision of how to understand risk and how to guide folks developing and guide folks testing, right? The MASVS for the mobile stuff, the OWASP ASVS, let’s say level two for the cloud APIs and the web infrastructure if that doesn’t just consume those endpoints. Testing of the local web services through conventional network or OWASP Top 10 maybe because that’s a lower exposure, lower threat surface.

John Verry: (31:45)
That’s what I’m thinking would be cool because I mean, that’s how we’re piecing it together. I’m looking at your document. I’m trying to see, are you giving… I look to your guidance because you guys are the experts. And I think we’re smart, but we’re not experts. So I’m looking at your guidance to say, are we doing stuff right? I’m looking in this document and I’m like, “I’m reading between the lines, but I’m not seeing that exactly.” You know what I mean?

Aaron Guzman: (32:10)
Like cohesively between each of the projects, you’re saying?

John Verry: (32:14)
Yeah.

Aaron Guzman: (32:15)
Yeah. I think that’s definitely a general OWASP problem. We’re putting out such great work, and I’m trying to collaborate with each of the groups. But I think even one kind of standard that’s missing would be cloud. That would help round it out, whether it be a focus on AWS or just general, the Terraform stuff. All that’s still very relevant. But I think it’s definitely known that there is some lack of communication between project leaders and the collaboration internally.

Aaron Guzman: (32:49)
But I think, yeah, there would be a good opportunity for us to figure out how we can create that narrative, right, for IoT specifically and how the web and mobile play. And then if we ever get a kind of cloud-based standard, I think it would really put that maturity in place. But I don’t think there’s a framework out there or an avenue right now to where… But I think it’s definitely a good idea. It’s just a matter of meeting-

John Verry: (33:23)
Time.

Aaron Guzman: (33:24)
Yeah, time, meeting amongst each other, and figuring out the best forward and what makes sense and what do readers and users… how can it be practical.

John Verry: (33:36)
Right. For anyone that’s looking at the screen, as most of the VS, verification standards, do, you break it down. You make it very, very simple to understand which controls apply at which level, right? Realistically, when we’re looking at this is level one is a lower risk level than level two and level three. Hence, level three is 124 controls. Level one, I think it was 60-something or something of that nature. I was specifically looking for one area.

Aaron Guzman: (34:05)
This is a draft, by the way, just for the listeners, the watchers.

John Verry: (34:10)
Okay.

Aaron Guzman: (34:11)
Yeah. I just saw the leveling off, and that’s something that we’re going to fix for the release.

John Verry: (34:18)
Okay, cool. Yeah. I apologize, yeah. I know it’s a draft, and you were kind enough to share it. And I appreciate that. Here’s where it was interesting to me is when I look at identity, identification authentication, when we do an MASVS test, it’s like you have a standard and you test that and you’re done. When I was looking at this, I’m looking at this and I’m thinking to myself, “Okay, identification and authentication, how many times might I use this iteratively within an IoT ecosystem?” Right?

John Verry: (34:44)
There’s device to device. There’s machine to machine, API to API. There’s the user authenticating to the web service running on the device. There’s the mobile app authenticating to the device, the mobile app authenticating to the cloud-based APIs.

Aaron Guzman: (35:00)
Yeah.

John Verry: (35:00)
When you look at some of this and the complexity and every device is different, every ecosystem’s different, so there is no one might have this, one might not. Do you see some of this stuff having to be done iteratively on a per authentication instance that we identify?

Aaron Guzman: (35:23)
I think we try to take the step back. Because if you see in 2.1.4, we talk about user services and starting a common framework. Because you’ll have your own way of authenticating to, say, Google Cloud, for example, IoT. Then you’ll have your own way communicating with the app, and they’re not necessarily talking to each other. You can bypass one or the other, which I’ve done before in assessment. I took some of that knowledge there and tried to put what was the biggest areas of authentication and identity, but right here we’re totally redoing this section. Individually and iteratively testing, it’s not a straightforward answer that I can think of, right?

John Verry: (36:10)
Okay.

Aaron Guzman: (36:10)
Yeah. I mean it’s a complex situation just because when we’re thinking about onboarding, zero touch onboarding or even zero anything, you know what I mean, and trying to think of being a step ahead to provide that guidance. Even for some of these, right, in order for you to have a secure identity, you have to have a root of trust, a hardware root of trust for only hardware can assert that identity. See, we had gotten some feedback from volunteers, and we’re taking that in for sure. Want to make it more clear and actionable and concise.

John Verry: (36:49)
One other question I had for you, so the communications are another area that’s really interesting to me. When you test families of devices, very often those devices in the world of digital landing management, they have their own communication protocols. You could have a device that’s talking wifi, Bluetooth, 6LoWPAN, and an 802.15 proprietary protocol. It was interesting to me that you specifically called out Bluetooth and wifi, and I understand they’re the most prevalent.

Aaron Guzman: (37:22)
Yep. That’s why.

John Verry: (37:23)
Yeah. Do you anticipate that you’ll have guidance for other protocols? Do you think that we should just have more of a general section for other protocols that are not these specific ones to make sure that we do have a way to address these?

Aaron Guzman: (37:44)
Yes. That’s what we opted to do is call out explicitly the most common protocols, and then in the general section-

John Verry: (37:51)
Oh, so this is what’s intended. This section here is intended to… If we were doing a project that had, let’s say, Zigbee, you would anticipate that the general section would be how we would test Zigbee?

Aaron Guzman: (38:04)
No. I wouldn’t say how you would test Zigbee, but what Zigbee should be built with. And I think there was one on anti-jamming. That’s not Zigbee. That’s Z-Wave, for example. I think Zigbee may have or maybe a newer iterative version of Zigbee, but we didn’t take a look at those as being the most prevalent, at least not right now. But I think if it makes sense in the future, we’re open to it.

John Verry: (38:30)
Yeah, no. Because I was thinking about the work we’ve been doing recently in applying this to that, which is why that one wireless one’s in there. Do you have plans for a testing guide at some point? Because there was some stuff in here that I think we know a decent amount, and a couple things I was like, “I’ve never heard of.” Like I haven’t looked it up yet, but Integrity Measurement Architecture.

Aaron Guzman: (38:51)
Yep.

John Verry: (38:53)
If somebody said, “Hey, prove this… ” I don’t know that I would know where to start with a test for that.

Aaron Guzman: (38:59)
Yeah. Did we include that?

John Verry: (39:04)
Unless I made it up.

Aaron Guzman: (39:07)
Maybe a software platform.

John Verry: (39:09)
Integrity, let’s see. Is it?

Aaron Guzman: (39:14)
I say that, I’ll give more context.

John Verry: (39:15)
There it is.

Aaron Guzman: (39:16)
Yeah. We did three, yeah.

John Verry: (39:17)
Verify that Integrity Measurement Architecture is in use and appropriately configured. Now it is L3, but I look at that and I was like, I made a mental note like, “I better look up Integrity Measurement Architecture before I jump on the call with Aaron.”

Aaron Guzman: (39:31)
Yeah. The reason why I say that is because in order for you to use an IMA kind of architecture, you have to have a TPM.

John Verry: (39:37)
Ah, okay.

Aaron Guzman: (39:38)
And you have to configure it a certain way with-

John Verry: (39:43)
Real quick, just to make sure I’m right and anyone listening, TPM is the Trusted-

Aaron Guzman: (39:48)
Platform Module.

John Verry: (39:50)
… Platform Module that we use for encryption, storing credentials, things of that nature?

Aaron Guzman: (39:55)
Yes, yes. It’s not as common for embedded devices because it’s a bigger chip and it’s more expensive. And you have to usually use the Trusted Computing Group, TCG. They have a TSS TPM library to open standard, open library. It’s not as widely deployed. Now looking at it and talking about it, I’m like, “I don’t know if we should actually include it because of that.” People ask about it, though. I know our customers ask if we have this in place. Essentially, what it does is measure every file on the file system. It takes a hash, and those hashes are stored in the TPM because it’s not modifiable by user space.

John Verry: (40:40)
Oh, that’s cool.

Aaron Guzman: (40:40)
So it’s an added layer. It’s actually-

John Verry: (40:45)
Almost like Tripwire sitting on my device, so it’s any file of import changes. Yeah, that would be nice. It’s funny that, so these TPMs-

Aaron Guzman: (40:55)
It’s kernel based, though.

John Verry: (40:57)
Oh.

Aaron Guzman: (40:57)
So it’s LSM.

John Verry: (41:00)
Okay. So is the implication of that that… I know on the resilience stuff with the MASVS, you’re looking to see whether or not you can alter kernel space, whether or not you can recompile the app and see if it detects that it’s been monkeyed with. Would the TPM or the IMA help with that?

Aaron Guzman: (41:22)
Yes, especially if you’re trying to, let’s say, run arbitrary code or something like that, it won’t let you.

John Verry: (41:30)
Okay.

Aaron Guzman: (41:30)
It won’t let you, yeah.

John Verry: (41:31)
Okay. That’s cool. Well, thank you for explaining that to me.

Aaron Guzman: (41:34)
Yeah. Like I said, it’s a complex even deployment model at scale, a lot of dependencies, a lot of things to think about.

John Verry: (41:42)
Cool. You have an incredibly unique perspective as an author for both the CSA and IoT Controls Framework as well as the ISVS. How would you compare and contrast them or when should somebody pick one or the other?

Aaron Guzman: (42:02)
Yeah. That’s a great question. I say it’s a great question because I feel like the CSA IoT Controls Framework is like a FedRAMP for IoT program level. There’s many different stakeholders. You have privacy. You have safety. You have the SRE infrastructure folks, the network folks. It’s really a holistic program and how you go about applying those controls into your program, your enterprise IoT program, as opposed to ISVS, which is a little bit more drilled in, a little bit more focused… Well, it’s a lot more focused on, again, measurements. How can you test whether this is in use or not, and even at a high level, some of it’s a binary yes or no.

Aaron Guzman: (42:57)
There’s some more specific requirements that go into a bit more detail, and we provide some examples. But if you’ve ever read anyone, I’m sure there’s some CISOs here who manage FedRAMP programs and some of the requirements there. You have to read between the lines and really try and make sense of it. Not to say that the IoT Controls Framework is something like that, but it’s more just at a program high level, a lot more stakeholders involved. It’s not something that is meant to be testable from a pen tester perspective or an assessment. It is from an architect level, though, how you’re going to build things.

John Verry: (43:43)
Gotcha. I’ll scope the numbers, right, because 8228 and 8259 from the NIST folks, right?

Aaron Guzman: (43:50)
Mm-hmm (affirmative). Yep.

John Verry: (43:50)
One of those is more high-level guidance, right? And the other one drills it down to next level. I think it’s 8228 is the higher level guidance. Is it almost in that ilk? It sounds almost in a way that you could use the CSA to build your IoT development programs encompassing all of the various areas of interest and objectives and outcomes that we might want. And then we could use the ISVS more specifically to give some more directly actionable guidance to our developers and to our testers?

Aaron Guzman: (44:28)
Yep, yep. Yeah, that’s a good way of explaining it. Also, there’s no really process requirements outside from threat modeling and the first chapter there is a little bit of process requirements that we have in there. But there’s certainly a lot within the Controls Framework. Those are a little bit difficult to test or just like, “Hey, let me see if you have that process in place,” type of thing.

John Verry: (44:53)
I think we beat the hell out of the list of stuff that I sent you over pretty good. Anything we missed? Anything that you want to just touch base on before we say goodbye?

Aaron Guzman: (45:03)
Sure, yeah. We communicate on the OWASP Slack, and we have our own channel. It’s project-IS-

John Verry: (45:10)
You jumped ahead on me, man. I was going to give you-

Aaron Guzman: (45:12)
Oh.

John Verry: (45:14)
Yeah. You’re a veteran of the show. You know the process and you screw it up anyway, Aaron.

Aaron Guzman: (45:21)
Oh, ready for a drink, that’s why.

John Verry: (45:24)
No, no.

Aaron Guzman: (45:25)
It’s afternoon, officially.

John Verry: (45:27)
No third appearance for you now. Real quick, one last question for you. What’s next for you? Where are you taking stuff? Beyond your book, what’s the next thing that we’re going to see out of you relating to this IoT stuff?

Aaron Guzman: (45:44)
I mean, yeah. Last year we put out three projects with the Firmware Security Testing Methodology, IoTGoat, and I’m blanking on the other one. I want to spend some time there and work on the next versions. And even for the IoT Controls Framework, we’re working on version three with Cloud Security Alliance already. So refining, updating things. Also, the Embedded Application Security Project, yeah, we’re starting that back up along with the IoT Top 10 as well. There’s a lot. There’s a lot going on, yeah.

Aaron Guzman: (46:22)
IoT Top 10, literally, this month we’re getting those going, calling for data, working with the Bunny platforms. If you’re a consultancy that’s listening and say you have some data that you can share on the OWASP IoT page, we’ll have a call for data that you can submit. So yeah, there’s a lot going on, but exciting year.

John Verry: (46:45)
I’m not surprised. Before we say goodbye, if somebody wants to participate in OWASP, if somebody wants to participate in any of the projects that you’re talking about, somebody wants to pop a question to you, what’s the best way for somebody to catch up with Aaron?

Aaron Guzman: (47:03)
Yeah. You could find me. I’m pretty easy to find. You could find me on Twitter, @scriptingxss. LinkedIn as well, same thing. Outside of that, within the OWASP Slack. As I mentioned, I’m on the IoT security channel, embedded app sec, the project ISVS. Reach out. Or even the GitHubs, if you want to contribute to any of the projects, file an issue, ping us. We’re happy to work with you and get you going. I think for folks listening who want to get involved and don’t know how, reach out to the project leaders.

Aaron Guzman: (47:38)
It’s all volunteer driven. There’s no fee or anything to get involved or even join OWASP Slack. Everything’s open, vendor neutral. Yeah, we won’t be able to function without volunteers and the help. We won’t be able to put out the content that we do without folks giving some of that energy and passion. So look forward to hearing from some of you. I think I had a few folks reach out from the last podcast that we did.

John Verry: (48:05)
That’s awesome, man.

Aaron Guzman: (48:05)
It was good.

John Verry: (48:06)
That’s awesome. Well, as always, sir, I appreciate your expertise and willingness to spend a few minutes here. I always feel a lot smarter after I get done chatting with you.

Aaron Guzman: (48:17)
Thanks, John. Appreciate it. What about the last question, though? You missed it last time.

John Verry: (48:20)
Yeah. You want to-

Aaron Guzman: (48:20)
You missed it last time.

John Verry: (48:23)
All right. Right. Are you ready?

Aaron Guzman: (48:24)
Yes, yes.

John Verry: (48:24)
All right. An amazing or horrible CISO. What fictional character or real person would make an amazing or horrible CISO and why?

Aaron Guzman: (48:32)
I think Shaq. Shaq would make an amazing CISO.

John Verry: (48:37)
Why would I expect nothing else from an LA guy? You know? I mean…

Aaron Guzman: (48:43)
He’d be like, “Let’s get things done.”

John Verry: (48:47)
You didn’t go Kobe.

Aaron Guzman: (48:49)
No, RIP.

John Verry: (48:51)
Yeah. But all kidding aside, all kidding aside, Kobe would be a good answer because he was a beast in life beyond-

Aaron Guzman: (49:00)
He’d be on it. He’d be-

John Verry: (49:00)
I don’t know if you saw some of the stuff he was accomplishing as a businessman after he left the court. He attacked his life outside of basketball he did his life inside of basketball. Pretty amazing guy. Shaq, I think physically, yeah. Can you imagine Shaq walking down to your desk and looking down at you and going like, “I want this done… ”

Aaron Guzman: (49:22)
By tomorrow.

John Verry: (49:23)
“… and I want it done by tomorrow.”

Aaron Guzman: (49:26)
Okay. Say less.

John Verry: (49:29)
What’s he, like seven-four, seven-three?

Aaron Guzman: (49:33)
I think he’s, I don’t know, seven, seven-one, something like that. He’s a big dude.

John Verry: (49:37)
Well, 350 pounds.

Aaron Guzman: (49:39)
Yeah, easy.

John Verry: (49:40)
Yeah, exactly.

Aaron Guzman: (49:41)
Ain’t messing with that.

John Verry: (49:41)
Well, that’s a great answer, man. All right. Listen, have an awesome weekend.

Narrator: (49:47)
You’ve been listening to The Virtual CISO Podcast. As you’ve probably figured out, we really enjoy information security, so if there’s a question we haven’t yet answered or you need some help, you can reach us at [email protected]. And to ensure you never miss an episode, subscribe to the show in your favorite podcast player. Until next time, let’s be careful out there.