The HackerOne Success Index - Hacker Breadth and Depth

Linus’s Law states, “given enough eyeballs, all bugs are shallow,” meaning broader and deeper testing populations find issues faster. This fifth blog in the HackerOne Success Index (HSI) series explores Hacker Breadth and Depth data from over 2,500 active hackers participating in hundreds of programs.

Our data show the best performing programs on HackerOne attract not only more overall hackers but more repeat hackers. Repeat hackers are responsible for the majority of resolved reports and bounties on the HackerOne platform. There is a clear upward trend in bounty amounts as a hacker submits more unique, valid reports to the same program. The more time a hacker spends looking at your software, the more valuable the reports are likely to be. For programs, there’s significant value in building hacker loyalty. For hackers, diving deep into the same code and building great relationships with security teams pays off.

Breadth and Depth Averages

The above graph shows the average monthly hacker participation across the different HackerOne programs -- all, top performers in this dimension, Public, and Private.

The average HackerOne program gets 16 participating hackers per month, and 6 of those are repeat hackers. The top programs attract an average of 56 hackers per month, with 19 repeating. The data also indicates that Public programs have better Breadth and Depth on average than Private programs.

What does it all mean?

• The most competitive programs attract about 50 new hackers per month, and retain around 20 month-to-month. Want to attract more hackers? Rewards, responsiveness, and providing new challenges through increasing programs scope are all ways to attract top hackers.

• More hackers make you more secure. Don’t believe us? Recent research demonstrates a very strong linear correlation between the number of participating hackers and the number of vulnerabilities fixed.[1] Hackers have diverse skills and approaches, and tend to explore different parts of in-scope properties.

• Go public! Public programs get more unique hackers and more repeat hackers. You have more control over a public program than ever with the new Signal Requirements feature.

Program Loyalty Pays

The chart above demonstrates that, repeat or “loyal” contributors account for the vast majority of the resolved reports on HackerOne, and an even greater portion of the bounties paid. This shows that repeat hackers are more valuable to programs, because that’s where the great majority of the bounty money goes. While Reputation is a good indicator of report quality and validity, the data show that multiple submissions to the same program are an even better predictor of validity and impact.

Bounty payments for hackers tend to go up on average after the first submission to a program, indicating the reports are more valuable.

Let’s look at the data. The above chart is generated by first selecting all HackerOne programs with more than 100 total bounties paid. Then, each hacker’s bounty is sequenced (1st, 2nd, 3rd, etc, submission) for each program. For each rank, we calculate the average amount and plot it. The results show that at an aggregated level the average payout increases as a hacker finds more valid bugs in a bounty program.[2]

Key Takeaways:

• For programs, retaining hackers increases the likelihood of finding high severity issues. Treat your hackers well, and reward them for their loyalty, and you can expect them to reciprocate with significant issues as they get to know your code.

• For hackers, it pays to develop a good professional relationship with programs. Most programs will steadily grow rewards as report quality and impact increase. Keep building great relationships with security teams, and it will generally prove worth it.

As always, we welcome any questions or feedback. Happy hacking!

-HackerOne Customer Success and Data Science team

[1] Mingyi Zhao, Jens Grossklags, and Peng Liu. "An empirical study of web vulnerability discovery ecosystems." Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security (CCS). ACM, 2015. http://sites.psu.edu/mingyi/wp-content/uploads/sites/11890/2014/04/An-Empirical-Study-of-Web-Vulnerability-Discovery-Ecosystems.pdf
[2] Thomas Maillart, Mingyi Zhao, Jens Grossklags and John Chuang. Given Enough Eyeballs, All Bugs are Shallow? Revisiting Eric Raymond with Bug Bounty Market. The 15th Annual Workshop on the Economics of Information Security (WEIS), 2016.

HackerOne is the #1 hacker-powered security platform, helping organizations find and fix critical vulnerabilities before they can be criminally exploited. As the contemporary alternative to traditional penetration testing, our bug bounty program solutions encompass vulnerability assessment, crowdsourced testing and responsible disclosure management. Discover more about our security testing solutions or Contact Us today.