August 4, 2017

Last Updated on January 12, 2024

There are dozens of companies that currently provide a Phishing Tests service (including Pivot Point Security :>)).  Why? Because we are all desperate for a solution to social engineering/phishing attacks—and it seems logical that phish testing should create greater awareness and greater awareness should reduce the success rate of phishing attacks.
But does it?

Sampling the Success Rates of Phishing Tests

I was curious, so I dug into the data from our last two years of phishing tests.

I looked specifically at cases where: 

  • We had phish tested the same organization at least twice; 
  • The gap between phishing tests was no less than 6 months and no more than 15 months, and; 
  • The organization did not use our (or another) security awareness training product. That is, they “self-educated” their users. 

The results were really interesting: 

  • During the initial test, 42.4% of the phish testing participants clicked on the phishing link in our email.  
  • During the second test, 39.1% of the test participants clicked on the phishing link in our email.  
  • For exactly 50% of the organizations, performance improved from Test 1 to Test 2. For the other 50%, performance went down. 

I don’t think anyone would argue with me characterizing those results as “underwhelming.” 
Digging a little deeper… The test results I referenced are from a more sophisticated attack that doesn’t just look at yes/no on the clicked link. Instead, once the link is clicked, it walks the user through multiple pages where increasingly sensitive information is requested on each page. I wondered if performance had improved in that regard.

Here are the results from the second analysis: 

  • During the initial test, 16.2% of test participants completed the process, thus providing sensitive business information that could have been directly used in an attack.  
  • During the second test, 18.8% of test participants completed the process, thus providing sensitive business information that could have been directly used in an attack. 
  • For 33.3% of the organizations, performance improved from Test 1 to Test 2. For 66.7%, it went down. 

Again, the results were highly underwhelming.  To be frank, this really surprised me.  I figured that, while it’s easy to click on one link without thinking, once the requests for sensitive information got higher and higher the participants’ “Spidey senses” would start tingling. Wrong.

The Verdict

So, what does this tell us?  Phish testing your employees does not work—at least not as a standalone technique to reduce your phishing risk. 
And that makes sense because the optimal use of a phishing test is not to educate, but rather to assess the effectiveness of the security awareness activities already taking place.  So arguably it was the education that failed, not the phishing testing.
Personally, I believe the right combination of Security Awareness Education with ongoing reinforcement will notably reduce phishing risk, and a well-executed phishing test can confirm that. 
We introduced our own Online Security Awareness Education product at the start of 2017 and should soon be in a position to assess my theory with data from the 10,000+ users that we already have licensing our program.
As soon as we have enough data I will publish a follow-up blog… Fingers crossed (for all of us)!