Case Study: Pro-active Log Review Might Be A Good Idea

Andrew Valentine
January 14th, 2013

With the New Year having arrived, it’s difficult not to reflect back on last year’s caseload. While the large-scale data breaches make the headlines and are widely discussed among security professionals, often the small and unknown cases are the ones that are remembered as being the most interesting from the investigators point of view. Every now and again a case comes along that, albeit small, still involves some unique attack vector – some clever and creative way that an attacker victimized an organization. It’s the unique one-offs, the ones that are different that often become the most memorable and most talked about amongst the investigators.

Such a case came about in 2012. The scenario was as follows. We received a request from a US-based company asking for our help in understanding some anomalous activity that they were witnessing in their VPN logs. This organization had been slowly moving toward a more telecommuting oriented workforce, and they had therefore started to allow their developers to work from home on certain days. In order to accomplish this, they’d set up a fairly standard VPN concentrator approximately two years prior to our receiving their call. In early May 2012, after reading the 2012 DBIR, their IT security department decided that they should start actively monitoring logs being generated at the VPN concentrator. (As illustrated within our DBIR statistics, continual and pro-active log review happens basically never – only about 8% of breaches in 2011 were discovered by internal log review). So, they began scrutinizing daily VPN connections into their environment. What they found startled and surprised them: an open and active VPN connection from Shenyang, China!  As in, this connection was LIVE when they discovered it.

Besides the obvious, this discovery greatly unnerved security personnel for three main reasons:

  • They’re a U.S. critical infrastructure company, and it was an unauthorized VPN connection from CHINA. The implications were severe and could not be overstated.
  • The company implemented two-factor authentication for these VPN connection. The second factor being a rotating token RSA key fob. If this security mechanism had been negotiated by an attacker, again, the implications were alarming.
  • The developer whose credentials were being used was sitting at his desk in the office.

Plainly stated, the VPN logs showed him logged in from China, yet the employee is right there, sitting at his desk, staring into his monitor. Shortly after making this discovery, they contacted our group for assistance. Based on what information they had obtained, the company initially suspected some kind of unknown malware that was able route traffic from a trusted internal connection to China, and then back. This was the only way they could intellectually resolve the authentication issue. What other explanation could there be?

Our investigators spent the initial hours with the victim working to facilitate a thorough understanding of their network topology, segmentation, authentication, log collection and correlation and so on. One red flag that was immediately apparent to investigators was that this odd VPN connection from Shenyang was not new by any means. Unfortunately, available VPN logs only went back 6 months, but they showed almost daily connections from Shenyang, and occasionally these connections spanned the entire workday. In other words, not only were the intruders in the company’s environment on a frequent basis, but such had been the case for some time.

Central to the investigation was the employee himself, the person whose credentials had been used to initiate and maintain a VPN connection from China.

Employee profile –mid-40’s software developer versed in C, C++, perl, java, Ruby, php, python, etc. Relatively long tenure with the company, family man, inoffensive and quiet. Someone you wouldn’t look at twice in an elevator. For the sake of case study, let’s call him “Bob.”

The company’s IT personnel were sure that the issue had to do with some kind of zero day malware that was able to initiate VPN connections from Bob’s desktop workstation via external proxy and then route that VPN traffic to China, only to be routed back to their concentrator. Yes, it is a bit of a convoluted theory, and like most convoluted theories, an incorrect one.

As just a very basic investigative measure, once investigators acquired a forensic image of Bob’s desktop workstation, we worked to carve as many recoverable files out of unallocated disk space as possible. This would help to identify whether there had been malicious software on the system that may have been deleted. It would also serve to illustrate Bob’s work habits and potentially reveal anything he inadvertently downloaded onto his system. What we found surprised us – hundreds of .pdf invoices from a third party contractor/developer in (you guessed it) Shenyang, China.

As it turns out, Bob had simply outsourced his own job to a Chinese consulting firm. Bob spent less that one fifth of his six-figure salary for a Chinese firm to do his job for him. Authentication was no problem, he physically FedExed his RSA token to China so that the third-party contractor could log-in under his credentials during the workday. It would appear that he was working an average 9 to 5 work day. Investigators checked his web browsing history, and that told the whole story.

A typical ‘work day’ for Bob looked like this:

9:00 a.m. – Arrive and surf Reddit for a couple of hours. Watch cat videos

11:30 a.m. – Take lunch

1:00 p.m. – Ebay time.

2:00 – ish p.m Facebook updates – LinkedIn

4:30 p.m. – End of day update e-mail to management.

5:00 p.m. – Go home

Evidence even suggested he had the same scam going across multiple companies in the area. All told, it looked like he earned several hundred thousand dollars a year, and only had to pay the Chinese consulting firm about fifty grand annually. The best part? Investigators had the opportunity to read through his performance reviews while working alongside HR. For the last several years in a row he received excellent remarks. His code was clean, well written, and submitted in a timely fashion. Quarter after quarter, his performance review noted him as the best developer in the building.

Comments

  1. This is nearly unbelievable. Is he overpayed or are the chinese underpayed? Crazy story indeed.

    Posted by: Stephan Schielke on January 15th, 2013 at 9:39 pm
  2. “Quarter after quarter, his performance review noted him as the best developer in the building.”

    Elvis has left the building.

    Posted by: JoeThePimpernel on January 15th, 2013 at 9:45 pm
  3. So, we’re saying Bob’s activity here was frowned upon? Cause I’m betting no one explicitly mentioned this to him in the interview.

    Posted by: James on January 15th, 2013 at 10:09 pm
  4. Pretty cool idea apart from the whole scary critical infrastructure component. Would this be more acceptable (and maybe even encouraged) if there were an NDA in place and the company dealt with comicbooks instead of critical infrastructure?

    Posted by: Bob on January 15th, 2013 at 10:26 pm
  5. Sounds like someone read “The 4 Hour Work Week” and decided to try it out…

    Posted by: Tony on January 15th, 2013 at 11:25 pm
  6. Sounds like a “4-Hour Workweek” success story.

    Posted by: mark zero on January 15th, 2013 at 11:39 pm
  7. Sooo… where’s the problem? He improved his personal profit and the quality and efficiency of his work, obviously. And all that by using standard business practices – get money to do the job, then pay someone else less to actually do it.
    This guy is an american hero and deserves a medal. I’d even go so far to call him a modern times Tom Sawyer, but since the chinese didn’t pay HIM to do his work, that’s just a little bit too far off.

    Posted by: Unterdosis on January 15th, 2013 at 11:47 pm
  8. ha ha ha smart guy hope they gave him a raise

    Posted by: emk on January 16th, 2013 at 12:23 am
  9. The worst part is his job meant something to the country. The best part we actuallu got the best out of China for cheap!

    Posted by: neil m on January 16th, 2013 at 12:41 am
  10. So this may have gone unnoticed for longer if the Chinese had come through a locally hosted VPS instead of making a direct connection?

    Posted by: Paul on January 16th, 2013 at 1:49 am
  11. Bob should have proxied the contractor’s connection through his home’s network.

    Posted by: Djilali on January 16th, 2013 at 1:52 am
  12. Gerät Job Bob!

    Posted by: Peter Schmitz on January 16th, 2013 at 1:58 am
  13. Surely that guy has to be promoted to manage a division that outsources sw dev to China. It’s the thing he is obviously very good at.

    Posted by: Alice on January 16th, 2013 at 3:13 am
  14. Sounds like they hired a lazy person to do a tech job. Perfect job because, according to Bill Gates, that is exactly what you want to do. “http://www.goodreads.com/quotes/568877-i-choose-a-lazy-person-to-do-a-hard-job” In typical fashion for tech people, he figured out the easiest way to do his job(s). Why is this news? It’s innovative, I’ll give him that. Sounds like he should be a high level manager instead of coding for a living. Maybe this is a clear example why most average people don’t understand tech people.

    Posted by: anon coward on January 16th, 2013 at 3:22 am
  15. So…. what was the name of the Chinese firm he was contracting with? They didn’t happen to list a phone number?

    Posted by: Tyler on January 16th, 2013 at 3:41 am
  16. So he managed a multi project setup, are able to communicate the requirements to remote “team members”, deals with different time zone, deliver in time, with good quality … Sounds like he was just in the wrong position.

    Posted by: ThoMo on January 16th, 2013 at 3:43 am
  17. Disturbing (and a little bit entertaining) has the bones for a Holywood script – would probably go straight to DVD unless they can swing a couple of big stars. Prob need to hold someone hostage to make it really sizzle.

    I would be interested to see how this played out from a employment / legal perspective. This chap was caught but how far reaching could this be in other organisations globabally.

    Posted by: Andrew on January 16th, 2013 at 5:21 pm
  18. Let me get this straight. When companies outsource employees, that’s just good management, but when employees outsource their companies, that’s bad? Bob deserves a raise, a promotion and a speaking tour for how to improve you personal profits.

    Posted by: GPW on January 16th, 2013 at 6:01 pm
  19. Sounds like the perfect candidate for service provider manager or third party account manager. It’s not easy managing third party oursourcing firms to develop something that works.

    Posted by: Fariz on January 16th, 2013 at 6:12 pm
  20. Djilali is right. Bob was wise enough to outsource himself, but not smart enough to figure out someone would eventually figure out a VPN connection from China was unusual. There were plenty of ways of avoiding this, but ultimately what he did would have been PERFECTLY legal had he been a sub-contractor himself. Then he could have kept his sub-sub-contracting hidden from the client. Alas, poor Bob, he hardly knew thees.

    Posted by: aragond on January 16th, 2013 at 6:48 pm
  21. Boy! Does this give me an idea! LOL, a one day work week, here I come , ,

    Posted by: Rocky on January 16th, 2013 at 6:55 pm
  22. So – whole companies can do this to the detriment of the US economy, but a single individual can’t? Noted.

    Posted by: Rich on January 16th, 2013 at 6:59 pm

Leave a Comment

*