Q&A: How Facebook handles product security, with help from outside bug hunters

Peter Oehlert is the director of product security for Facebook, leading the social network’s efforts to ensure the security of the apps and features used by more than 1.5 billion people around the world.

A former Microsoft security software engineer with experience in consulting and startups, he stopped by the GeekWire offices this week for a conversation about security practices at Facebook, the security landscape for startups, the role of white-hat hackers in keeping Facebook secure, and more.

Listen to a podcast of the conversation below, and continue reading for a transcript.

Todd Bishop: You’re actually in Seattle for a conference, an FTC conference. What was the message that you were getting out there earlier today?

Peter Oehlert: I participated in FTC’s Start With Security event and really was talking to new startups in terms of companies that were creating different kinds of technology and trying to give them a little bit of advice about how they can embed security into the fabric of their company and into the products that they’re building.

Todd Bishop: It’s an interesting topic because so many startups are just trying to get their minimum viable product out in the market and a lot of times I would imagine that security is the last thing on the checklist.

Peter Oehlert: Yeah. It’s really true that a lot of companies, I think, tend to think that way. I think a lot of companies have been successful starting up that way in the past, but the threat environment really is changing and now it’s not so much that you can start a company that collects data and not be at risk until you get bigger, like it used to be. Really, these new startups are entering the risky internet environment just like everybody else.

Todd Bishop: What were your biggest tips for them, if you had a couple of takeaways?

Peter Oehlert: Sure. I think a couple of takeaways were really to make sure that everybody really takes the responsibility of securing their product. It’s not just about the developer or the engineer who’s building it, but it really is the responsibility of everybody in the company, especially when the startup is small. Another key takeaway is that just starting the process is really important. Carving off some time on a regular basis, even as you’re building your minimum viable product, is important to make sure that you’re incorporating security in from the beginning.

Todd Bishop: Are there common tools out there that startups can use? It seems like that is the common practice of startups. They look to other people who have built the tools that they can use. Are there things that Facebook does or other vendors do that people can lean on when they’re building new apps?

Peter Oehlert: Yeah, absolutely. There’s a number of tools that we talked about at the summit. On the panel with me was a gentleman, a former colleague of mine from Microsoft, who was talking about a couple of the tools that Microsoft has released under their Security Development Lifecycle. In terms of the specifics and the specific frameworks that we talked about, we talked about a lot of startups who are really focused on a particular application stack, and within that stack, it will have some specific security properties. The frameworks might have a security implication — it’s really just about asking the question. What is the security property of this thing which I’m thinking about building my application on? That can really help startups get going.

Product security at Facebook

Todd Bishop: Got it. You’re the Director of Product Security at Facebook. You’ve worked at Microsoft in the past. You’ve been a consultant and you’ve also been in a startup of your own. What is your job like now? Give us the scope of your responsibility and what you cover at Facebook.

Peter Oehlert: Sure. One of the things I was really excited about Facebook is the breadth of things that Facebook is doing, and so within the product security space, it gave me a lot of opportunity to have a really big impact. In terms of my regular day, I tend to spend a lot of time looking out for where the team is going and making sure that, overall, the organization is always focusing on application security when we’re building applications so that we have really resilient products that we’re creating.

Todd Bishop: What are the differences between security that you’re facing, the vulnerabilities and the threats that might exploit them on Facebook’s platform compared to operating systems of old? Windows? What are the major similarities and differences?

Peter Oehlert: I think a lot of the similarities are that the same kinds of processes and techniques that we used to look for security vulnerabilities, to fix security vulnerabilities are agnostic to the technology that they’re being applied to. They work as well at Facebook as they do at Microsoft as at lots of other companies.

The things that really end up being different are really about the technology itself. The kinds of vulnerabilities that we would face on a web application are just going to be a little different than what you might have in an operating system scenario. The kinds of vulnerabilities that we’re looking for and how we go about fixing them is going to be a little bit different, but a lot of the actual steps that we take to identify those vulnerabilities and get them fixed are the same.

Todd Bishop: What are the biggest threats that you’re seeing right now? What types of things? Is it people exploiting code? Is it social engineering? Can you give me the landscape of a bad stuff out there?

Peter Oehlert: Sure. Security is a really broad topic and there’s a lot of different ways in which things can happen. I think with regard to product security, the stuff that I really focus on, a lot of the stuff that you would see on a website like Facebook actually looks not that different from most many other website. There’s a great organization called OWASP that publishes something called the OWASP Top 10 and it’s basically the top 10 worst web application vulnerabilities. A lot of those are similar for other companies as they are for Facebook.

One of the great things that Facebook has been able to do is create secure frameworks that we can use to eliminate rather than one issue at a time focus on eliminating an entire class of vulnerability. The number one issue on the OWASP Top 10 Cross-Site Scripting is actually something we don’t have to deal with very much anymore because we’ve created a great framework for eliminating it making it really difficult for any of our developers to actually introduce that vulnerability.

Todd Bishop: What is cross-site scripting? I’ve heard of that before but it’s not like a classic like buffer overflow or something like that.

Peter Oehlert: It’s different than a buffer overflow. Cross-site scripting is a webpage vulnerability where the application might treat user input as code and so what ends up happening is an attacker can send code up to the application and get it interpreted to do whatever the attacker said. What in effect happens is the attacker is able to introduce their own logic into the application as it’s running and that lets them do all kinds nefarious things.

Todd Bishop: Could somebody potentially use that or another technique to gain control of a user’s computer via Facebook? Is that a potential vulnerability?

Peter Oehlert: It’s not usually so much control of the user’s computer. A lot of web application security vulnerabilities are limited to being able to exploit things that are in the browser, so it’s not so much that the vulnerability would allow it to read data out of your Quicken file, but a lot more that it would let the attacker do anything that they wanted on Facebook as the victim.

Todd Bishop: So much of our lives happen inside the browser now that it is the operating system for the web or I guess maybe that’s not the perfect analogy, but it’s very much where people exist now, the same way used to be inside the OS.

Peter Oehlert: Yeah. They used to be a much nicer barrier having the browser as this buffer to keep people from operating on the machines, but these days most everything that people care about is out on the internet in a cloud service somewhere and so it makes it more important to get all those fixed.

Todd Bishop: You work a lot with Facebook’s internal engineers, making sure that the products they’re building are secure. When a new Facebook engineer comes on, what are the key things that you might drill into them that might be different from what they would have done at a prior company?

Peter Oehlert: Sure. I think training is certainly an important part of how we engage with engineers and the Product Security team leads the training course that all of our engineers take in the boot camp which is our onboarding process for engineers. The things that they really learn are, one, really trusting in the frameworks. Unlike, I think, a lot of startups that don’t necessarily have the resources, Facebook has invested a lot of money in creating a number of really great frameworks that can make writing security vulnerabilities really difficult to do. This way, developers can focus on what they actually want to achieve rather than having to think as they write each line of code whether or not there might be a vulnerability present there.

The other really big thing that we emphasize is that the Product Security team is there to really help work with them and collaborate. Facebook’s security team isn’t the security team that goes around telling people they can’t do stuff. We really work hard to collaborate with our engineers and say, “That’s a great idea. Here’s how we can do it securely,” rather than saying, “No, don’t do that.”

A surprising bug

Todd Bishop: What’s the most surprising bug or attack you’ve ever seen? You’ve been at Facebook for about year now?

Peter Oehlert: For about a year, yeah.

Todd Bishop: What’s the thing that made you go, “Whoa,” or just maybe kept you up for 24 hours trying to resolve it?

Peter Oehlert: I’m trying to think of some of the different attacks. I think of one that was maybe surprising only in that it’s something that we have a great framework to protect against that in a lot of ways it shouldn’t have happened because we have this technology already built. When we launched Messenger.com we got a number of White Hat reports. We haven’t talked about White Hat, but the Bug Bounty Program that the Product Security team also runs. We got a number of reports through that bug bounty program that there was what’s called a cross-site request forgery. Just this particular vulnerability in our website.

Because we had frameworks to make sure that this didn’t happen, it was quite surprising that we had this brand new product that we’re releasing that had this vulnerability in it. It turned out that our understanding of how our frameworks were connected to some of the other things that people were doing was a little less complete. Even though a couple of security people had looked at it, we just missed the fact that there was a missing flag. It was really as simple as one line of code. It was a great point to remind us that even when we have great frameworks we have to make sure that they’re being used correctly.

Todd Bishop: Interesting. There is still a human element in all of this.

Peter Oehlert: It’s code and there’s so many different things you can make code do if you want to. I’m not sure that we’re ever going to get to a point that we won’t have humans at the end of the process taking a look at things and making sure that things work together. We work hard to build a lot of tools to help us scale up or impact so that we can have a really broad impact, but at the end of the day, there’s a lot of thinking involved in what we do.

Todd Bishop: You’ve worked at Microsoft, and Facebook, and other companies. How would you compare the culture of Facebook to the other places where you’ve worked?

Peter Oehlert: Sure. I think one of the things that surprise me a little bit was how much security is embedded into the culture at Facebook. What I really came to the conclusion about is that Facebook’s mission is about connecting people, connecting everybody around the world, and that as part of that mission it really flows through the culture.

Where other companies might have a pretty strict release process with gates that have checkboxes to make sure that security has been done correctly at this point and security has been done incorrectly at the next point, Facebook works a lot more about connecting people, connecting the security team with the engineers. Trusting the engineers to take responsibility for what they’re building and really understand the implications and how important protecting and being good custodians of all of our users’ data is. It works well for us and I think a lot of it just comes down to that very big mission which we’ve set in front of ourselves and how much that’s trickled through the entire organization.

Todd Bishop: Correct me if I’m wrong here, but there was such a series of high profile, big consumer data breaches in the early part of 2015. Am I wrong in this impression — it feels like at least in terms of the big headlines that that there’s been a little bit of a lull over the past few months. I’m trying to think back … maybe we’ve just become desensitized to it. I don’t know.

Peter Oehlert: Yeah. In terms of the big consumer breaches, it’s hard to say exactly how often they come. I know that I definitely a bunch of them. I can’t remember how many letters I got last year of people having lost my data, but it was, at least, six. It might have been more.

Todd Bishop: As somebody in security, what do you think when you get one of those notifications?

Peter Oehlert: I’m usually frustrated. From a consumer standpoint, it’s very much like ‘I gave you my data, and you went, and you lost it.’ … You have very little control over it, so something that’s totally out of your control and something which you’re not really responsible for somebody has gone and lost it.

The other thing I think about in terms of the broader industry is for the companies that are doing well in this space how important it is that they work with other industries and people that are a little bit newer to the technology space or to running some of these operations. How much responsibility we have to help bring them up. I think Facebook has done that in a number of ways. There’s industry collaborations. We do things like talk to startups about how they can embed security into what they’re doing.

I talked a bit about the frameworks that we build. We’ve opened source to a number of those frameworks to allow others to mimic some of our successes so that they can adopt the frameworks that we built and have some of the same benefit.

Todd Bishop: Speaking of open source, are you thinking of things like Threat Exchange and OSQuery? What are those?

Peter Oehlert: Threat Exchange is a platform that built that allows different companies to share information about the threats that they’re seeing. It’s a little bit different than what the Product Security team is doing. We’re really focused on how we’re building stuff. Threat Exchange is a little bit more focused on actually running and operating these data centers with these machines, with these corporate assets, and if I get a threat and somebody else down the street is also seeing the same threat, then maybe the rest of the ecosystem can learn from those threats and actually move faster to close whatever holes might exist. The idea is really as a herd we have a more immunity against these malicious attackers than if we each hide what’s actually going on.

In terms of OSQuery, OSQuery is a tool that we found that we really needed ourselves. We have a lot of machines that we operate and we operate them on some platforms that don’t have … some platforms have really good ability to collect information and some don’t have as much. OSQuery is really meant to address that deficiency in Linux and Mac systems in terms of being able to collect all the configuration — what’s running on the machine, how it’s configured, how the user accounts have been set up, and pull all of that data together so that we can enforce the right policy on that.

It was really successful internally. It was something we opened source and it’s one of the most popular security projects on GitHub. It’s definitely been really gratifying to see the community both adopt it. In terms of our open source projects, it has one of the most number of outside contributions. The community has really picked it up and is going forward with it, which I think really just says it fills a big need that people had.

Tips for Facebook Users

Todd Bishop: It is Safer Internet Day. I realize this is not your core expertise. Your focus is on the product security and making sure that Facebook’s developers are building secure products, but for the average Facebook user out there, what are the key things that they should know and keep in mind, things to do, things not to do on Facebook?

Peter Oehlert: One of the great ways that consumers can protect themselves on Facebook is using a tool called Security Checkup, which allows them to review their settings, how they’re sharing data, what data they’re sharing publicly, what data they’re only sharing with friends. They can very quickly and easily get an understanding of exactly what it is that they’re sharing on Facebook.

Another great one is a tool called Login Approvals, which provides an extra check when you’re logging in to make sure that it’s really you that’s logging in to your Facebook account and not somebody else. On top of the Login Approvals we also send notifications to your email account that’s registered with Facebook so that if you do buy a new Android phone and you log in with it, you might get an email from Facebook that says, “Hey, we saw that somebody was logging in from an Android phone we don’t recognize. Was that really you or was that somebody trying to get into your account?”

The last thing I would mention for consumers is broader than just specific to Facebook, but passwords are really important. I know that it’s difficult, I know that it’s hard to have different passwords, but it is really important to have different passwords for the different sites that you use. You shouldn’t have the same password for your bank. It’s for Facebook, it’s for some random company that you bought something on from the internet with that one time. Using different passwords is a must. There’s some great tools called password saves, password stores. There’s a number of them that are out there. What they allow you to do is create an individual password for each of those different places.

I’ve worked hard with my own family to convince them that this was the right thing to do. After you get into the habit of it, it actually works out pretty easily, but more than that, when you get those letters in the mail about how somebody’s lost your data, you can feel certain that if you only gave them one password, that it’s not shared with any other system, whoever it was that got your account data isn’t going to be logging into your bank or to your Facebook account and stealing your money or posing as you.

Todd Bishop: Anything else you’d want to get across that we haven’t touched on yet about Facebook, security, just things in your realm and things that people out there should know?

Peter Oehlert: You know, we haven’t talked too much about the White Hat program, which I think is a really exciting program that Facebook does. Facebook has been running a bug bounty, we call it the White Hat bug bounty, for the last 5 years. We’ve paid out over $4 million, had something like 2,500 different bugs that people have submitted to us. The way it works, basically, is it’s engaging with the security community around the world. They have the permission to come and try to find security vulnerabilities in our product, and if they submit them to us and they’re valid, then we will pay them. We’ll give them money for the vulnerabilities that they found.

Those White Hat reports are a really valuable source of information for us about products that we’ve already released, but more than that, it’s a way to deeply engage with the security community and even help bring them up. One of, I think, the unanticipated side effects of running the program is the impact that we have some of these people’s lives.

We recently learned that one of our researchers has been using the bounty money that they’ve gotten from Facebook to fund their education. We know that some other bounty researchers will actually use the accolade, will publish their name saying that they helped find a bug in Facebook to land job offers. Around the world, we’re able to really enable a whole new set of people that maybe aren’t going to show up in any of your room here in the U.S. because that can be very complicated, but it can still help us from around the world and get a really big benefit out of it themselves.

Todd Bishop: Do you ever end up hiring folks onto your team from the White Hat program?

Peter Oehlert: We absolutely do. We’ve hired a handful of folks from the bug bounty program and, in fact, a number of them are still working on the Product Security team operating the bug bounty that they once were submitting to. In fact, the highest bounty that we’ve ever paid was $33,000 and change and the person who won that bounty is now one of our employees.

Todd Bishop: What was the bug?

Peter Oehlert: The bug was a remote code execution in a system. It was a pretty important bug. It would have allowed an attacker to essentially run code on one of our servers, and so it was really great that it was found by a White Hat, and not exploited by somebody more nefarious. We paid him a lot of money. He had a good history of really great bugs so we’re like, “We need to hire this guy.”

Todd Bishop: I would imagine that the trick is to pay them just enough to keep them motivated, and to be willing to come, and to work for you, and get an actual salary, but not so much that they’re off on the beach for the rest of their life.

Peter Oehlert: No. Actually, that does seem like one optimization that we might make, but it’s not one that we’ve really chosen to make. We really try hard to make sure that all of our payouts are as fair and as reasonable as possible. Because we can’t really share the payment decision process with everybody, it means that people have to trust us a little bit, and so we work really hard to make sure that we’re consistent and that we’re even-handed.

A lot of times when there’s something in doubt, when there’s a question, we’ll always air on the side of paying somebody more. We’ve even had bugs where somebody submitted something and we’d paid them out, and a little bit later we figured out that impact was actually a lot bigger than anybody recognized, and we’ll go back and pay them more.

The one other thing I would give a call out about the bug bounty, it’s a great way to practice your security skills. If you’re interested in application security, you think you might have an act for it, you can read some stuff, read blogs, read books, that’s great, but actually putting fingers to keyboard and giving it a try, that’ll help you understand what you really know and we will pay you.

Todd Bishop: This is a very fundamental question, but how can you tell when someone is testing your systems for good versus evil? In other words, do they have to register first and say, “Hey Facebook, don’t go after me if I infiltrate your system”?

Peter Oehlert: The way the terms are set up is that the activity that they do has to be in good conscience, but there are certainly times where internal controls might identify something automatically, and there might be something that a researcher was trying to do that was prevented by our internal control systems, and we don’t pay them out for that. We’re already protected against that. Really the question we want to have answered by researchers is can they find out a way around those things.