High-Severity Catastrophes

My last post was about the dangers of not mitigating low-severity vulnerabilities. Now I want to look at the high-severity findings that are not worth remediating.
Most responsible people would like to see these remediated. After all, they’re highly severe! Frequently, though, the cost to remediate a high-severity finding is too great to justify the security benefit. There could be any number of reasons for this; below is a hypothetical example.

Our Protagonist

William Ford is a newly minted director of information security at TCC, a mid-sized call center. He started out as a call center representative at a previous employer but made a nominally lateral move into IT because of his knack for computers and desire to not get screamed at on an hourly basis.
His promotion from IT whiz kid to information security professional happened as a part of the natural order of things: TCC got hacked and, as a system administrator, Will took it personally. He started putting in extra hours implementing a real patch routine, implementing an IDS, and tightening the organization’s firewall rules. The nights he didn’t spend at the office he spent reading about security and listening to podcasts.
Soon after Will was named Director of Information Security and Compliance, TCC got their first network penetration test. This came at the request of a very large prospective client. It was hinted that this prospective client is the reason why Will’s new title included the word ‘compliance’--and that the client’s demands helped Will get his promotion in the first place.

The Report

Will tore into the Penetration Test report with aplomb, skipping twelve pages of executive summary and administrivia to get to the thick forty page section titled “Technical Details,” an itemized technical treatment of every insecurity that the consultants could find. As he read, Will jotted a list of todos: patch an errant UNIX server, reconfigure a web service. He skipped around the document, finding similar vulnerabilities and grouping them together, sketching out the course of action necessary to fix the problems.
One particular cluster of problems that flummoxed Will involved TCC’s phone system.
The report listed several problems with the phone system: insufficient access control, easily guessable password, et cetera. The centerpiece was a finding titled vaguely “Lack of Protection of Sensitive Data (in Storage and Transit).”

The System

Here’s what Will knew about the phone system.
About three years ago, TCC replaced its analog phone system with a digital VOIP system. If you haven’t worked with them before, know that call centers have very special phone systems. Most corporations need a phone on every desk, an extension for every employee, and a voice mail system. Call centers have additional needs which necessitate specialized technology.
In addition to the typical office PBX, TCC uses an automated phone dialer that can blend incoming and outgoing calls. Modern dialers are predictive, meaning that they maximize capacity by predicting when outbound agents are seconds away from completing their calls and dial the next number in advance, shaving a few seconds off each call. These systems also have to manage complex compliance rule sets so that, say, calls made to California don’t happen after 8:00PM Pacific time. They have to take in various data from external sources, including do-not-call and unsubscribe lists, and translation tables for area code changes (which happen much more frequently than you’d imagine). The dialer collects gobs and gobs of statistics for reporting. Aside from the dialer, the call center reps also have an electronic time clock that they have to punch in to at the beginning of every shift, and some other workflow management program that they use on a minute-by-minute basis. The specific type of workflow management system depends on whether they are telemarketers, bill collectors, surveyors, charities, political campaigners, or some other type of person you don’t want to talk to. Call centers also usually have some kind of war-room screen showing average wait time, wait queue length, number of CSRs online, et cetera. All this stuff has to work together somehow. Older systems are usually held together with duct tape and prayers. Newer systems are sold as one massive, expensive “turnkey” solution.
(Aside: I hope you have this image now. If not, do this. First, do an image search for the term “Call Center”. See all those smiling, young, multi-ethnic faces framed by expensive starched collars and cyberpunk headsets? Okay, now look at the technology that supports them: . That’s an awfully big chunk of IT for a vertical whose organizational core competency it is to be pleasant with you on the phone.)
To what extent does this system store sensitive information? Will had no idea. He suddenly realized that he had been staring at his todo list for so long that his coffee turned cold. He had no idea how to even begin thinking about fixing this stuff.
In Will’s experience, fixing any chunk of technology is easy as long as you had sufficient knowledge about that technology. He clearly didn’t know enough about this stuff since he didn’t know how to fix it. The one person in TCC who would know about the phone system was Susan, the server administrator.

Meet the System Administrator

Will maintained good rapport with Susan so he decided to elicit her opinion over a few slices at the local NY style pizza place. He specifically wanted to now about protection of sensitive data, or the lack thereof.
Between nibbles, she delivered an infodump on the call recording subsystem. It was protected by a shared password. Anyone who knew the password could log in and listen to calls, download them as .WAV files, or delete them. The system had a thick client interface that would allow for searching and browsing based off of date, time, phone number, or representative name, but this interface was only accessible by connecting to the system over remote-desktop. Also, the calls were stored on disk as .WAV files on disk in big folders, immediately accessible once a user connected remotely. The recording system was sold as a “network appliance,” mainly because of the impressive storage requirements: it had six 500GB EIDE drives. The sales literature boasted RAID-5 but the system showed a full 3TB capacity --“So do the math on that,” Susan said. “Oh, and get this: it has to be a CONCAT. It’s at near-full capacity all the time: the call recording software implements a sort of ring-buffer for retaining files without overflowing the disk.”
“What about backing up?”
“Ha. It takes more than 24 hours for a full network backup. I don’t even know when one completed successfully.” She grimaced. “Oh, and if you try to apply Windows patches, the vendor says you’ll lose support. I don’t know what PCI says about all that, but it sounds pretty pooched to me.”
Susan gave Will more bad news than he was asking for. The report didn’t say anything about backups and RAID problems. All the same, she wasn’t able to offer much in the way of advice. Will was ready to meet his penetration testers. Maybe they’d have better advice for how to fix the system.

Meet the Penetration Testers

A few days later Will met up with the pen testers for the read-out of their report over lunch at a local steak house. He showed up early to discover that they were already there drinking straight whiskey.
The pen test team consisted of two fellows, Russ and Dave, who comprised one half of their consulting company, Attack Hawk Security. Over rare porterhouse cuts of steak, the friendly and capable penetration testers described the impact of this vulnerability.
Dave, the younger member of the team, started to explain how credit card numbers could be exfiltrated from the call recording system. It integrated directly with the VOIP network, effectively functioning as a network tap that sniffed all VOIP packets off the network, storing and indexing them as appropriate. Anyone who knows or can guess the password (“123456,” for the record) could install some very expensive audio file voice analysis software, chew through the .WAV files converting them to text one-by-one, and use grep.exe to pull out any fourteen-to sixteen-digit numeric strings. Bob is your uncle.
Russ went on to explain—off the record, he added—that if TCC wants to start using this system for PCI data they should get ready to grab their ankles. PCI mandates that any system that stores, processes, or transmits credit card data is in scope. This includes call recording systems, as long as they can be queried. Unfortunately, TCC’s prospective client dabbled in credit card data.
Their report’s remediation advice was vague: encrypt sensitive data in storage and protect it with role-based access control. Will asked for some elaboration. "Well, the encryption has to be strong," Russ said with a discerning frown. "AES-128 or better."
"Of course, but how do I put encryption into a network appliance? I can’t just sprinkle crypto-fairy dust around the rack in the server room."
"This is not your fault. Your vendor screwed up.” Russ smiled, “You’re just responsible for it is all. So talk to the vendor about it," said Russ as he signaled for the check. “It would also be worthwhile to see what else is on the market that can fulfill this business requirement."
With that, Attack Hawk climbed into their DeLorean and blasted off toward the sunset.

Meet the Vendor

Will put in a call to his sales representative. The rep was nice and only spent about the first twenty minutes angling for an up-sell; then he got very defensive.
Any sort of point-to-point transport encryption would cripple the call recording system, since it listens to all the calls going across the wire.
The vendor offered other call recording solutions--but nothing with RBAC built in, and nothing with encryption. The sales rep pointed out that they don’t advertise PCI compliance as a sales feature.
“Under no uncertain terms, if you were to try to integrate an call recording solution from a third party--even if something exists that can be jammed in place—we would not be able to honor your support contract,” said the rep. For perspective, he then reminded Will that TCC puts in a support ticket about once every two months.
Regarding security patches, the rep called shenanigans: “We roll up all platform-level patches with the software updates that we send out once every six months. You get the Windows patches, you just wait a little longer for us to regression test them first.”
“Okay, let’s take a step back. What would you recommend I do to secure your product then? Or should I just turn to one of your competitors?”
“I dunno. I’m in sales, not security. I will tell you though that the sort of advanced security features that you’re looking for aren’t implemented by anyone that would consider us a competitor. It’s true that there are call center solutions out there that offer these features. They target a different market than we do.”
Will needed to see Sarat, CFO of TCC, about funding for a new call system. Before he did, though, he needed to make sure his case was bulletproof. Was the system really all that bad? Will had to see for himself. Even if it was, there had to be a cheaper way out of this. What if TCC just stopped using the call recording system all together?

Meet “the Business”

Will put time on a call center manager’s calendar.
Next morning, Will met with Mitch, one of the call center managers. Mitch’s office was decorated with sketches made by his children and vaguely Christian motivational posters. His desk and credenza supported half a dozen stacks of papers, interleaved with as many empty Chick-fil-A sandwich envelopes. He verified a lot of what Susan said and even ran Will through his use of the call recording system.
"Mitch, what if we don’t do any call recordings?"
"That’s not even funny. TCC has contractual obligations with every existing customer to record calls for quality purposes. It’s not even boilerplate: those recordings are there to support service level agreements for customer satisfaction, not to mention legal reasons," Mitch explained.
"We need call recordings to defend ourselves in court from consumers who claim my reps verbally assaulted them. And then on the rare occasion that a CSR does cross the line, we need those call recordings even more. Think about it: we’re going to fire the CSR and they’re going to come back at us with lawyers. We need the recordings to show that they were let go for cause."
That sealed it for Will. If TCC wanted the extra business, they’d have to replace the call system.

Meet the CFO

This was first time Will had met with a CFO before and he was going to assemble a PowerPoint presentation, but since the meeting got chopped by forty minutes, Will just punched up a few bullet points as a “recommended agenda” in an email.
The meeting didn’t really adhere to that agenda though. Sarat listened attentively to Will for about four minutes. Then shit got real.
"Without getting too deep into how corporate finances work, understand that the company had to secure an additional line of credit four years ago to get the new phone system," Sarat explained. As he spoke, he put his hands in this professional-looking Star-Trek-communicator shape out in front of his chest. Sort of like steeple-fingers, but with the fingers pressed together on each hand. "Having this much credit is financially risky. We were only able to make this purchase because we convinced our investors that their old system would no longer support our growing company."
"The investors believed us because the old system had been online for a long time. Ten years, in fact. It was installed the second year that the company was in business. That said, it was still a hard sell. We had to show that we were replacing it in a financially responsible manner. The new phone system has a ten-year amortization schedule."
Seeing that Will didn’t entirely grasp what an amortization schedule is, Sarat helped him along. "That means that we promised our investors that this new, expensive, technologically advanced system would take care of our needs for at least another ten years--as long as the old system."
Sarat folded his hands in front of himself. “Now to do what you want on the schedule you want, I would have to call an emergency meeting of the board of directors. I would have to explain that the investment decision we made four years ago was unwise because the new system does not meet these security requirements. These requirements did not exist then, so I’m not sure how that part of the conversation would go. Doesn’t matter. I would then explain that we need to take this phone system to the dumps--even though we’re still paying it off--and secure another line of credit to purchase an even newer system that WILL meet these requirements and, hopefully, support our growth into the financial processing market which was not something we were planning on doing until just now.”
“Alternatively, we could just pass on this contract." Sarat pantomimed this by swooshing his hand in a horizontal semi-circle with a steady angular velocity. Then he flicked his wrist under and around, drawing two fingers up in a presto-change-o maneuver, “or even better, we could find a way to win this contract with our existing phone system. Is this something you think you can help us with?”
“I sure hope so,” said Will.
Sarat’s eyes were already off Will and he was clicking through his inbox. “That’s why we hire only the best security personnel.”

Meet the Client

In the end, there was nothing Will could do. TCC’s system had irreparable security deficiencies that they could not afford to fix. Will kept busy enough with his other responsibilities, and fixing the rest of findings, but he set aside time each day in his calendar: “Plan Call Recording Soln.” Every day for one hour, he sat in thought.
By the time Will started to get pulled into talks with the client, he felt helpless. Like a caged animal. His first call was a teleconference with IT directors from both companies. His involvement was perfunctory; no one asked questions of him, no one assigned him action items.
Will’s second call came a week later, it was one-on-one with Sharif, a senior security analyst from the client. Sharif had read the copy of the report he received from Attack Hawk. During the call Will tried not to be too angry, defensive, or exasperated. He described the fixes already in place. When Sharif asked about the unprotected sensitive data in transport, Will said matter-of-factly that he did not have a plan in place yet, but he was open to advice. Sharif acknowledged this just as matter-of-factly, and moved on to other matters.
When the call ended, Will felt relieved. It was like he expected to get shot in the gut, but he wasn’t bleeding.

Denouement

More teleconferences were had. Will had a few technical calls with the client’s security staff. Some of them were kind hi-how’s-your-father conversations. Others, Will felt like he was in the firing line. Sarat also pulled him into some high-level discussions.
The “high-severity” finding stayed open. TCC kept its phone system. Will implemented some stronger network isolation, cutting off access between the phone system and the PC network. Susan got a new fiber-channel backup system. To show her appreciation, she employed a hack of mind-blowing elegance to configure the call recording system to auth against their corporate Active Directory system. The client gave a sizable chunk of business to TCC, but not the PCI-relevant work. Will, along with about a dozen other key personnel got a little hemispherical glass paperweight commemorating the win.