Dissecting /etc/passwd

Discussing local file inclusion with with @pacohope and @kseniadmitrieva, it became evident that I have more than a few non-obvious things to say about the value of /etc/passwd when in the hands of an opportunistic adversary. It is available on most (all?) Unixes, yet it is rich in idiosyncrasy. I threw together my thoughts and put them here.

If you know of more interesting details, drop me a comment.

Sometimes Nmap Takes Down a Datacenter

"We don't perform denial of service testing," I say to my client.
Those words exit my mouth with an alkaline sting. Lies, all lies! We detect DoS by looking at version strings in server banners. There's no deep kung fu in this; Nessus has plugins for it. And if I see that my client's server could be brought down with killapache.pl, I will let them know. The truth is that our tests can occasionally identify DoS conditions safely, without actually denying any service.
But even this is only a half-truth. The whole truth is that even safe, industry accepted pen testing practices can wreak havoc on a network.
I was an information security officer for a respectable organization during a short lived run on the IPv4 address space market. Fortunately, mere weeks before network managers started jumping off the roofs of their data centers, one of my network guys snagged a few Class-Cs and had them pointed them at our DR site for safekeeping. Fat, dumb, and happy we were the doomsday preppers of the 32-bit address space apocalypse.
Time went by and we started gearing up for our annual external penetration test. There had been lots of changes in our network that year and I thought it would be best to scope the assessment by looking up our public IP assignments in ARIN, cross-referencing it against a scripted mass dig(1) of all the externally accessible DNS names in our asset management system. Everything checked out, a few externally hosted marketing sites notwithstanding.
The first night of testing went off without a hitch.
Eleven o'clock at night on the second night I got a call from the on-call system administrator, a bit miffed because the DR site had been taken down. The entire data center had gone dark! I quickly phoned my tester and had him pull the plug. Before you could say Ctrl-C, I had hung up and put the call back in to my system admin. Everything was back up.
I didn't expect that our pen tester pulled down our site. I only had him pull the plug for political reasons. If something went wrong, the networking team would blame the pen testers. Even when no tests were scheduled, they'd call me up to ask if there were any unscheduled scans going on. It's a defense mechanism they'd learned from years of being the infrastructure whipping boy. For years, every time a tablespace fills up, a process falls into an infinite loop, or a memory board smokes, people would run to the networking team.
At that place and time though, penetration testing was the new black magic. No one understood it; everyone feared it. If the lights flickered or the cafeteria ran out of chicken tikka masala, pen testing was probably the reason. So I had to put a stop to the testing before the real incident response could begin. I didn't expect that my lone tester could bring down a whole data center by himself though. The site coming back online mere seconds after he stopped scanning was too much of a coincidence. I called him back up and explained the synchronicity.
"I don't understand. All I was doing was an Nmap."
"Send me the command you ran." This breaks protocol a little. As a paying customer it is my responsibility to wag my finger across the phone at the tester and demand an assurance that this will never happen again. He would respond by categorically denying any wrong doing. Then he'd spend the rest of the engagement running a single thread discovery scan. The end result would be a blank report and a mysteriously fragile network.
Instead, I like to treat these nightmare inducing outages as a chance to learn something about the world. His Nmap scan seemed legit. No crazy plugins, no elaborate timing. It was a simple TCP connect scan. There were no payloads. I had him run it again while I sniffed the network at our border.

In front of our firewall was a border router who's only job was to be a DMARC between us and our ISP. It was a 1U implementation of a very simple bit of logic:

for each packet
  if packet.destination.ip_address ∈ $respectable_org.ip_addresses
  then
    send packet to $respectable_org
  else
    send packet to $isp
  end if
end for

Thing is, we had not yet added our new Class-C's to $respectable_org.ip_addresses. But our ISP's router implemented similar logic, and they had included the subnets in their list of our IP addresses. And of course I had included it in our tester's scope.
So what happened? Our tester started scanning our DR site on the second night, (the first night he had scanned production.) Our ISP sent each of his probes to our border router. Not recognizing the destination address as one of ours, the router sent the packet back out the port it came to our ISP. Our ISP recognized the destination address as belonging to us and send the probe right out the port it came, back to us. This two-node route loop lasted until the time-to-live on each packet decremented all the way--about thirty times per packet. And since the loop was as tight as the next hop away, it was much more impactful than running thirty times as many threads.
The penetration tester was exonerated, the appropriate routes were put in, and testing resumed. Ironically, it was the networking team's fault this time, though I'd hardly blame them.
The truth is, non-destructive means can reveal weaknesses that destructive attackers can exploit, non-destructive means can come to destructive ends with the most innocuous configuration problems, and that there's always the risk of something bad happening.
And to tell the whole truth, my client doesn't want to hear all of that. They want the assurance that their business won't suffer for having hired me. If I can't assure them of that with fewer than a dozen words, I can't assure them of that. So I'm stuck dragging out that old chestnut,
"We don't perform denial of service testing."

Proactive Unix System Maintenance for Security

I have an article up over at WServerNews, an online newsletter for Windows sysadmins. My article is a collection of anecdotes, each with a little moral lesson at the end.

Lest anyone think I'm a Windows bigot, I wanted to follow up that article with some quick prescriptive guidance for Unix system administrators.

Opportunity 1: Apply Software Updates

Ever since Windows for Workgroups, Windows SAs have been leagues ahead of Unix SAs in patching. This is born out of necessity: Windows systems have to manage floors of buggy workstations that get viruses daily. Your Unix systems don't give you as mature a tool set as they have.

All good system administrators know how to use rpm, pkgadd, apt-get or whatever to bring a system up to current. Figure out (and document) how to roll back an update for a system or even just a package. Write a script that reports the last date a system was patched and incorporate it into your system monitoring solution.

You might even want to learn how to use Puppet or cfengine to automate configuration changes.

Opportunity 2: Review Logs

Unix admins have grep and grep goes a long way, but when grep doesn't cut it anymore, you will want to stand up a log server like splunk or a SIEM like OSSIM.

Opportunity 3: Change Default Passwords

This is pretty universal. Read your software docs and check the password lists here and here.

Consider the Costs

My previous two posts covered two dysfunctions I see when people try to fix their security problems: ignore the low-severity issues, and try to fix massive high-severity issues and hit a wall. Not trying to fix a low should be the polar opposite of trying in vain to fix a high, but they share a common cause: they both ignore the cost of action.
Information security professionals have different methods for determining and describing the severity of a finding. When we use NIST 800-30, we eyeball a high/medium/low value based on our perception of impact and likelihood. When we use CVSS we think about the vulnerability in terms of exploitability and impact, dividing each into different factors to which we apply weights and modifiers. Adam Shostack has suggested that security professionals describe risk purely in terms of dollars. When we use severity in this way, we’re really trying to describe the benefit of remediation.
The process of remediation takes time. In most cases, money too. These are scarce resources. The science of deciding how to use scarce resources to achieve desirable ends is called Economics, and it has a tool we can use to help decide whether to remediate vulnerabilities: cost benefit analysis.
Simply put, the process for cost-benefit analysis is as follows: find out all the costs, find out all the benefits, and compare. If the cost is greater than the benefit, don’t do it. You can expound upon this as you like, accounting for stakeholders and externalities, comparing alternate projects by their cost-benefit ratios. But even bad CBA is better than none.
Enterprise IT is usually very good at cost-benefit analysis. If you have a PMO in your organization, ask them about it.
If you are responsible for the security of an organization, keep the costs in mind. If you are an assessor you can help by understanding your client’s business as best you can—especially their change processes—and look for cost effective solutions.

High-Severity Catastrophes

My last post was about the dangers of not mitigating low-severity vulnerabilities. Now I want to look at the high-severity findings that are not worth remediating.
Most responsible people would like to see these remediated. After all, they’re highly severe! Frequently, though, the cost to remediate a high-severity finding is too great to justify the security benefit. There could be any number of reasons for this; below is a hypothetical example.

Our Protagonist

William Ford is a newly minted director of information security at TCC, a mid-sized call center. He started out as a call center representative at a previous employer but made a nominally lateral move into IT because of his knack for computers and desire to not get screamed at on an hourly basis.
His promotion from IT whiz kid to information security professional happened as a part of the natural order of things: TCC got hacked and, as a system administrator, Will took it personally. He started putting in extra hours implementing a real patch routine, implementing an IDS, and tightening the organization’s firewall rules. The nights he didn’t spend at the office he spent reading about security and listening to podcasts.
Soon after Will was named Director of Information Security and Compliance, TCC got their first network penetration test. This came at the request of a very large prospective client. It was hinted that this prospective client is the reason why Will’s new title included the word ‘compliance’--and that the client’s demands helped Will get his promotion in the first place.

The Report

Will tore into the Penetration Test report with aplomb, skipping twelve pages of executive summary and administrivia to get to the thick forty page section titled “Technical Details,” an itemized technical treatment of every insecurity that the consultants could find. As he read, Will jotted a list of todos: patch an errant UNIX server, reconfigure a web service. He skipped around the document, finding similar vulnerabilities and grouping them together, sketching out the course of action necessary to fix the problems.
One particular cluster of problems that flummoxed Will involved TCC’s phone system.
The report listed several problems with the phone system: insufficient access control, easily guessable password, et cetera. The centerpiece was a finding titled vaguely “Lack of Protection of Sensitive Data (in Storage and Transit).”

The System

Here’s what Will knew about the phone system.
About three years ago, TCC replaced its analog phone system with a digital VOIP system. If you haven’t worked with them before, know that call centers have very special phone systems. Most corporations need a phone on every desk, an extension for every employee, and a voice mail system. Call centers have additional needs which necessitate specialized technology.
In addition to the typical office PBX, TCC uses an automated phone dialer that can blend incoming and outgoing calls. Modern dialers are predictive, meaning that they maximize capacity by predicting when outbound agents are seconds away from completing their calls and dial the next number in advance, shaving a few seconds off each call. These systems also have to manage complex compliance rule sets so that, say, calls made to California don’t happen after 8:00PM Pacific time. They have to take in various data from external sources, including do-not-call and unsubscribe lists, and translation tables for area code changes (which happen much more frequently than you’d imagine). The dialer collects gobs and gobs of statistics for reporting. Aside from the dialer, the call center reps also have an electronic time clock that they have to punch in to at the beginning of every shift, and some other workflow management program that they use on a minute-by-minute basis. The specific type of workflow management system depends on whether they are telemarketers, bill collectors, surveyors, charities, political campaigners, or some other type of person you don’t want to talk to. Call centers also usually have some kind of war-room screen showing average wait time, wait queue length, number of CSRs online, et cetera. All this stuff has to work together somehow. Older systems are usually held together with duct tape and prayers. Newer systems are sold as one massive, expensive “turnkey” solution.
(Aside: I hope you have this image now. If not, do this. First, do an image search for the term “Call Center”. See all those smiling, young, multi-ethnic faces framed by expensive starched collars and cyberpunk headsets? Okay, now look at the technology that supports them: . That’s an awfully big chunk of IT for a vertical whose organizational core competency it is to be pleasant with you on the phone.)
To what extent does this system store sensitive information? Will had no idea. He suddenly realized that he had been staring at his todo list for so long that his coffee turned cold. He had no idea how to even begin thinking about fixing this stuff.
In Will’s experience, fixing any chunk of technology is easy as long as you had sufficient knowledge about that technology. He clearly didn’t know enough about this stuff since he didn’t know how to fix it. The one person in TCC who would know about the phone system was Susan, the server administrator.

Meet the System Administrator

Will maintained good rapport with Susan so he decided to elicit her opinion over a few slices at the local NY style pizza place. He specifically wanted to now about protection of sensitive data, or the lack thereof.
Between nibbles, she delivered an infodump on the call recording subsystem. It was protected by a shared password. Anyone who knew the password could log in and listen to calls, download them as .WAV files, or delete them. The system had a thick client interface that would allow for searching and browsing based off of date, time, phone number, or representative name, but this interface was only accessible by connecting to the system over remote-desktop. Also, the calls were stored on disk as .WAV files on disk in big folders, immediately accessible once a user connected remotely. The recording system was sold as a “network appliance,” mainly because of the impressive storage requirements: it had six 500GB EIDE drives. The sales literature boasted RAID-5 but the system showed a full 3TB capacity --“So do the math on that,” Susan said. “Oh, and get this: it has to be a CONCAT. It’s at near-full capacity all the time: the call recording software implements a sort of ring-buffer for retaining files without overflowing the disk.”
“What about backing up?”
“Ha. It takes more than 24 hours for a full network backup. I don’t even know when one completed successfully.” She grimaced. “Oh, and if you try to apply Windows patches, the vendor says you’ll lose support. I don’t know what PCI says about all that, but it sounds pretty pooched to me.”
Susan gave Will more bad news than he was asking for. The report didn’t say anything about backups and RAID problems. All the same, she wasn’t able to offer much in the way of advice. Will was ready to meet his penetration testers. Maybe they’d have better advice for how to fix the system.

Meet the Penetration Testers

A few days later Will met up with the pen testers for the read-out of their report over lunch at a local steak house. He showed up early to discover that they were already there drinking straight whiskey.
The pen test team consisted of two fellows, Russ and Dave, who comprised one half of their consulting company, Attack Hawk Security. Over rare porterhouse cuts of steak, the friendly and capable penetration testers described the impact of this vulnerability.
Dave, the younger member of the team, started to explain how credit card numbers could be exfiltrated from the call recording system. It integrated directly with the VOIP network, effectively functioning as a network tap that sniffed all VOIP packets off the network, storing and indexing them as appropriate. Anyone who knows or can guess the password (“123456,” for the record) could install some very expensive audio file voice analysis software, chew through the .WAV files converting them to text one-by-one, and use grep.exe to pull out any fourteen-to sixteen-digit numeric strings. Bob is your uncle.
Russ went on to explain—off the record, he added—that if TCC wants to start using this system for PCI data they should get ready to grab their ankles. PCI mandates that any system that stores, processes, or transmits credit card data is in scope. This includes call recording systems, as long as they can be queried. Unfortunately, TCC’s prospective client dabbled in credit card data.
Their report’s remediation advice was vague: encrypt sensitive data in storage and protect it with role-based access control. Will asked for some elaboration. "Well, the encryption has to be strong," Russ said with a discerning frown. "AES-128 or better."
"Of course, but how do I put encryption into a network appliance? I can’t just sprinkle crypto-fairy dust around the rack in the server room."
"This is not your fault. Your vendor screwed up.” Russ smiled, “You’re just responsible for it is all. So talk to the vendor about it," said Russ as he signaled for the check. “It would also be worthwhile to see what else is on the market that can fulfill this business requirement."
With that, Attack Hawk climbed into their DeLorean and blasted off toward the sunset.

Meet the Vendor

Will put in a call to his sales representative. The rep was nice and only spent about the first twenty minutes angling for an up-sell; then he got very defensive.
Any sort of point-to-point transport encryption would cripple the call recording system, since it listens to all the calls going across the wire.
The vendor offered other call recording solutions--but nothing with RBAC built in, and nothing with encryption. The sales rep pointed out that they don’t advertise PCI compliance as a sales feature.
“Under no uncertain terms, if you were to try to integrate an call recording solution from a third party--even if something exists that can be jammed in place—we would not be able to honor your support contract,” said the rep. For perspective, he then reminded Will that TCC puts in a support ticket about once every two months.
Regarding security patches, the rep called shenanigans: “We roll up all platform-level patches with the software updates that we send out once every six months. You get the Windows patches, you just wait a little longer for us to regression test them first.”
“Okay, let’s take a step back. What would you recommend I do to secure your product then? Or should I just turn to one of your competitors?”
“I dunno. I’m in sales, not security. I will tell you though that the sort of advanced security features that you’re looking for aren’t implemented by anyone that would consider us a competitor. It’s true that there are call center solutions out there that offer these features. They target a different market than we do.”
Will needed to see Sarat, CFO of TCC, about funding for a new call system. Before he did, though, he needed to make sure his case was bulletproof. Was the system really all that bad? Will had to see for himself. Even if it was, there had to be a cheaper way out of this. What if TCC just stopped using the call recording system all together?

Meet “the Business”

Will put time on a call center manager’s calendar.
Next morning, Will met with Mitch, one of the call center managers. Mitch’s office was decorated with sketches made by his children and vaguely Christian motivational posters. His desk and credenza supported half a dozen stacks of papers, interleaved with as many empty Chick-fil-A sandwich envelopes. He verified a lot of what Susan said and even ran Will through his use of the call recording system.
"Mitch, what if we don’t do any call recordings?"
"That’s not even funny. TCC has contractual obligations with every existing customer to record calls for quality purposes. It’s not even boilerplate: those recordings are there to support service level agreements for customer satisfaction, not to mention legal reasons," Mitch explained.
"We need call recordings to defend ourselves in court from consumers who claim my reps verbally assaulted them. And then on the rare occasion that a CSR does cross the line, we need those call recordings even more. Think about it: we’re going to fire the CSR and they’re going to come back at us with lawyers. We need the recordings to show that they were let go for cause."
That sealed it for Will. If TCC wanted the extra business, they’d have to replace the call system.

Meet the CFO

This was first time Will had met with a CFO before and he was going to assemble a PowerPoint presentation, but since the meeting got chopped by forty minutes, Will just punched up a few bullet points as a “recommended agenda” in an email.
The meeting didn’t really adhere to that agenda though. Sarat listened attentively to Will for about four minutes. Then shit got real.
"Without getting too deep into how corporate finances work, understand that the company had to secure an additional line of credit four years ago to get the new phone system," Sarat explained. As he spoke, he put his hands in this professional-looking Star-Trek-communicator shape out in front of his chest. Sort of like steeple-fingers, but with the fingers pressed together on each hand. "Having this much credit is financially risky. We were only able to make this purchase because we convinced our investors that their old system would no longer support our growing company."
"The investors believed us because the old system had been online for a long time. Ten years, in fact. It was installed the second year that the company was in business. That said, it was still a hard sell. We had to show that we were replacing it in a financially responsible manner. The new phone system has a ten-year amortization schedule."
Seeing that Will didn’t entirely grasp what an amortization schedule is, Sarat helped him along. "That means that we promised our investors that this new, expensive, technologically advanced system would take care of our needs for at least another ten years--as long as the old system."
Sarat folded his hands in front of himself. “Now to do what you want on the schedule you want, I would have to call an emergency meeting of the board of directors. I would have to explain that the investment decision we made four years ago was unwise because the new system does not meet these security requirements. These requirements did not exist then, so I’m not sure how that part of the conversation would go. Doesn’t matter. I would then explain that we need to take this phone system to the dumps--even though we’re still paying it off--and secure another line of credit to purchase an even newer system that WILL meet these requirements and, hopefully, support our growth into the financial processing market which was not something we were planning on doing until just now.”
“Alternatively, we could just pass on this contract." Sarat pantomimed this by swooshing his hand in a horizontal semi-circle with a steady angular velocity. Then he flicked his wrist under and around, drawing two fingers up in a presto-change-o maneuver, “or even better, we could find a way to win this contract with our existing phone system. Is this something you think you can help us with?”
“I sure hope so,” said Will.
Sarat’s eyes were already off Will and he was clicking through his inbox. “That’s why we hire only the best security personnel.”

Meet the Client

In the end, there was nothing Will could do. TCC’s system had irreparable security deficiencies that they could not afford to fix. Will kept busy enough with his other responsibilities, and fixing the rest of findings, but he set aside time each day in his calendar: “Plan Call Recording Soln.” Every day for one hour, he sat in thought.
By the time Will started to get pulled into talks with the client, he felt helpless. Like a caged animal. His first call was a teleconference with IT directors from both companies. His involvement was perfunctory; no one asked questions of him, no one assigned him action items.
Will’s second call came a week later, it was one-on-one with Sharif, a senior security analyst from the client. Sharif had read the copy of the report he received from Attack Hawk. During the call Will tried not to be too angry, defensive, or exasperated. He described the fixes already in place. When Sharif asked about the unprotected sensitive data in transport, Will said matter-of-factly that he did not have a plan in place yet, but he was open to advice. Sharif acknowledged this just as matter-of-factly, and moved on to other matters.
When the call ended, Will felt relieved. It was like he expected to get shot in the gut, but he wasn’t bleeding.

Denouement

More teleconferences were had. Will had a few technical calls with the client’s security staff. Some of them were kind hi-how’s-your-father conversations. Others, Will felt like he was in the firing line. Sarat also pulled him into some high-level discussions.
The “high-severity” finding stayed open. TCC kept its phone system. Will implemented some stronger network isolation, cutting off access between the phone system and the PC network. Susan got a new fiber-channel backup system. To show her appreciation, she employed a hack of mind-blowing elegance to configure the call recording system to auth against their corporate Active Directory system. The client gave a sizable chunk of business to TCC, but not the PCI-relevant work. Will, along with about a dozen other key personnel got a little hemispherical glass paperweight commemorating the win.

Low Severity Vulnerabilities

Many technology and software organizations maintain a dangerous policy for remediating security assessment findings: "We fix the highs and mediums. Everything else can wait."

I recently conducted a penetration test a web application for a client who themselves did security assessments. This was an important application for the client. Functionally it was little more than a CRUD application, allowing users of various privilege levels the ability to create, read, update, and delete highly sensitive data. Mine was the fourth such assessment that this version of the application had received and after three rounds of remediation the client expected to pass easily.

My results? I uncovered one medium-severity finding and about a dozen low-severity findings.

Was the client disappointed? Only a little. They weren’t expecting the medium. The lows they had known about, in some cases for years. Their application has been accumulating them, each round of penetration tests adding one or two before their routine push to production.

There are a few reasons why this is dangerous.

Vulnerabilities combine with each other. When they do, their impact combines in ways which are not additive or even multiplicative.

Chris Gates is an expert at finding vulnerabilities in enterprise software. His phrase for this property is "Low to Pwned," and he has collected a number of case studies here.

One of the examples he offers is of a JBOSS installation with an exposed server-status URL and GET request parameters that include the session token. Both defects are commonly ranked as low severity. When they appear together in the same system an attacker can easily hijack any current session by reading the server-status page and copying those session tokens.

In his talk Gates demonstrates several of these problems and shows how they can be exploited with high business impact. Here's a recording of the talk he gave at hash days, it's well worth a watch.

This problem is compounded where vulnerabilities are scored with CVSS. The CVSS scoring guidelines clearly state “vulnerability scoring should not take into account any interaction with other vulnerabilities.” This guideline ensures that CVSS will give a lower score to all vulnerabilities that interact with any other vulnerability.

You only mitigate vulns with a CVSS score of 7+ and I only exploit vulns with a score below 7. I win. You lose.

— Info Security Jerk (@infosecjerk) November 10, 2013

Another danger is that penetration tests don't uncover every defect.

Consider an application that is deemed by risk management to have marginal value to the organization. Perhaps it's a blog that caters to an industry segment of the customer base. Marketing says that it drives sales and no one cares enough to disagree with them. No PII, no intellectual property, it's not even in the same DMZ as everything else. Penetration testing finds the following: "Session cookie with promiscuous domain and path settings" and "Missing HttpOnly Attribute in Session Cookie." Both are low-severity. The first is even a justified business requirement, as the blog's administration login is tied to SSO.

These two lows don’t combine in any interesting way. What happens if this site has reflected cross-site scripting along with a promiscuous cookie scope and JavaScript-accessible cookies? An attacker who discovers what a time-boxed penetration tester did not can whip up an attack that sends SSO tokens to an external server. The next effect is that the outside attacker gets an SSO session token. I wonder if webmail is tied to SSO....

Another danger is that severity ratings are estimates. By definition then, about half the time they are underestimates.

Penetration testers frequently don’t have a complete threat model to work off of. Once, a tester who worked for me wrote up a vulnerability that involved some misconfigured sudo rules on a system for which only administrators had logins. This vulnerability allowed one administrator-level role to access the files of another administrator-level role. At the 10,000 foot-view of visio documents, this is a non-issue. Just to make sure though, I stopped by the cubicle of system administrator who was responsible for the misconfiguration and asked ‘what happens if these guys can read your files? Is that a low-severity finding?’ He explained that no-no-no, it was a high-severity finding.

I recommend that organizations remediate those vulnerabilities that are worth fixing. This is a vague recommendation that I intend to flesh out later. For now, let it be known that some low-severity vulnerabilities are worth fixing.