Bad Penny » Presentations

Threats, Threat Modeling and Analysis

Ian Gorrie — Fri, 21 May 2010 17:30:41 +0000

This is a super high level presentation about basic threat modeling, SDL, and why a proactive stance is better than a reactive. I thought that it was fun.

Threats, Threat Modeling and Analysis

I’ve had even less time to myself than usual lately so let me apologize in advance for not separating and expanding on my speaking notes from the deck like I have in the past. To make up for it, please feel free to use this deck if you are introducing SDL to your team(s).

Download file formats:
pdf
keynote

The Art of Keeping Things Done

Ian Gorrie — Sat, 10 Apr 2010 00:32:04 +0000

The current field of information security is largely one of arcana, vagueness, arbitrary views, philosophy, mountaintop sages, a general lack of reliable data, and legions of vendors selling “best practices.”

It was my hope that I could help out a little by giving a talk on my take of how our industry can best navigate during these turbulent and weird times and come toward relevance and transparency.

That’s enough of a preface. Here’s the talk I gave at the Seattle NAISG meeting this month.

Opening slide

I started this talk with a brief clip of the last two minutes of Bruce Potter’s opening to Shmoocon 2010.

The whole video is available here. Other Shmoocon media is available here. My poor quality two minute clip can be found here.

Do you find this kind of talk discouraging? Do you take it personally? I do.

This is what I said when I was watching the streams.

Is this really the state of data in the information security industry?

So Bruce says that we’re “Failing at our jobs every day.” Does he has a point. I think that he does.

Largely information security, and at times IT in general, has not been relevant or contributing to IT goals. We’ve been largely thought of as a cost center or a hole that you have to dump money into to avoid not being in compliance and possibly going to jail.

Since we have been unable to have effective communication with business and larger industry, we have spawned the technology compliance industry as a byproduct of our failures to have a meaningful dialogue and back up our assertions with data.

Half of the problem here, I believe, is one of basic terms. Some people I have met think these things mean the same thing. Some other people are attached to some kind of security warrior monk philosophy where they are honor bound to defeat all insecurity and risk no matter the cost.

This is not how the world works. This is not how a successful risk management program works either.

It is about what is in the best interest and practical means of the organization.

There has been some unnecessary drama in PCI lately. The crux of the disagreement on conference panel discussions and elsewhere really boils down to this:

Compliance is not a governance model.

PCI exists because without it, some environments would not take the minimum steps to secure their data. PCI is a whip to those lagging behind on the bell curve industry average.

I have been informed recently and from a quality source that the average PCI assessment costs between 250k and 500k on average. These funds are straight out of IT and/or security budgets. These are funds that could be used to improve a security program instead of spinning the hamster wheel of pain.

So in effect, PCI punishes security programs that are already at the minimum standard and causes even greater problems for leaders the information security space who are being proactive and doing the prudent thing in that, depending on the QSA firm/consultant retained, there may be a disagreement on the compensating controls in place or visionary risk management decisions made.

But compliance has improved things, right?

Sadly, no.

A paper by Forrester Research, commissioned by Microsoft and RSA, the security division of EMC, found that even though corporate intellectual property comprises 62 percent of a given company’s data assets, most of the focus of their security programs is on compliance with various regulations. The study found that enterprise security managers know what their companies’ true data assets are, but find that their security programs are driven mainly by compliance, rather than protection. — Threatpost

In short: we are protecting the wrong things and we know it. Why are we doing it this way? Because we have failed to have a relevant conversation on risk as an industry for too long, now others are doing it for us.

Additionally, PCI is now considered a stream of revenue not only to auditors, but to the card issuing industry in general.

Think about what this means.

QSAs pick the fruit from the money tree, but the roots are the card issuers. The tree is going to get bigger and its fruit heavier.

So let us review the hamster wheel and its many problems.

We should be proactive, not reactive. We should lead the discussion with the rest of IT in what the data means and what to do about it instead of who is to blame for a gap in an audit map.

I boldly contend that no hamster wheel effort is a governance program as it is detached from the other processes at work. If an auditor is finding systematic flaws in your governance program, something is very wrong.

Please post your disagreements in the comments.

Process should yield something. The result of an information security program should be an increasingly favorable risk position, not a new process to keep everyone busy as a cost center.

A risk management program should not enforce the status quo. It should produce data and discussions should be based on that data when it is new and not at the next quarterly, yearly, or root cause failure meeting.

If you are not being proactive by designing tests for development, finding configuration and application errors, and assessing your threat and architecture landscape, you are not running a governance program. You are likely only compliant.

Focus on what is possible, not what is allowed. Do not overly rely on any one mechanism or technology to protect you. Test or evaluate each piece of your architecture (defense in depth is good plug here) on its own. Better yet, find a way for it to prove that it is working. Collect this data for your compliance people and those whos work product has generated what it is measuring.

No one should have a problem with data in itself. If rewards are given to those trying to figure out who to blame for problems instead of correcting the problems themselves, something isn’t working right.

Other toothless and unsupported maturity models and governance frameworks are not much better off than just relying on arbitrary standards and compliance efforts. They need someone to have their back and have real consequences baked in.

Risk management is the yin to the yang of quick-deploy-and-fix-later-maybe philosophy.

This is the same fight that quality assurance had twenty years ago and won. We have the same battles to make on the very same ground. All of the statistics about security flaws in software and systems are out there and undisputed; bugs are inexpensive to fix inline with development and orders of magnitude more expensive to fix later on. Choosing a fundamentally insecure architecture to base your business on and then using piecemeal efforts to mitigate risk after the launch is also a pretty bad, but common, idea.

The business decision is the weighing of the risk of opportunity to get to market first and viability of the business due to flaws after launch. To feed this decision, we need to give the business straight forward information and not snake oil, fear, doubt, or frantic hand waving.

Frameworks at least put leadership for security issues at the table instead of a project footnote, but is it enough?

We need more data, to be credible based on this data, and we need to be backed by executive leadership based on our credibility and data.

We need to stop being the philosopher sages of IT and start having actual justifications for the methods and solutions we, as an industry, are advocating employing.

If we don’t do these things, how do we know if we’re doing a good job?

We need to collect and share data.

Part of the big compliance discussion has been the argument of “they were breached, so they must not have been compliant at time of incident.”

What do you say to that if you don’t have a lot of data backing up your risk management decisions?

Some schools of risk management dismiss all measurements as arbitrary and worthless. I don’t see how they can call themselves risk managers at all unless they base their decisions on at least the attempt to take a proactive stance by measurement and estimation instead of the baseline of the minimum standard of not being provably negligent.

Not surprisingly, there is a variety of opinion even on this.

Mike’s argument in favor of Donn and mountaintop sages.

~~Adam’s~~Alex’s argument against mountaintop sages.

Also Alex’s talk about why we’re hosed having to pick between the two (and more).

There is a lot to win by being in a leadership position in reducing the number of flaws and inefficiency in an environment.

Here are some more wins.

..and a few more.

Much of this is based on an ITIL model and the IT Process Institute’s findings which they would like to sell you.

We had better figure this out soon before our environments get too complex for us to manage or assess. If we’re not there already, we’ll be there soon.

I contend that part of risk management is the ability to simplify and optimize. Do things for a reason and have some data to justify it. Don’t just to things because some other people you talked to at a conference once said it was a good idea or because it was in a magic quadrant in a leadership document you bought from someone else.

I thought that this was a good quote. Here’s a lot of Kabay’s work.

We, as an industry, have really talked about this for a very long time without much achievement. Most of the commercial product space hasn’t been interested and we haven’t made them be interested.

Most talks I hear stop there. So what do we actually do about it?

Not only should we transparently collect and base our decisions on data, but we should do it in a way that doesn’t make us look like a bunch of egotistical babies.

Work with people to improve things instead of Conan the Barbarian approach to program management; use the carrot instead of the stick. Help fix problems instead of just complaining how everything is trash and broken. Make some friends instead of beating them over the head with the compliance hammer.

Make things better. We can do it.

Here are some new sources of traditional metrics:

You should be aware of them because people talk about them a lot. They might not be very useful for you, but at least you’ll have something to talk about.

Things that are generic to the entire IT world may not be interesting to the place where you are working. If it’s not interesting to people in your realm, they are likely useless.

I think I totally lifted these two slides from a Metricon talk as I completely don’t talk this way. You should read all of the Metricon talks. They are all interesting and we don’t hear enough of this kind of talk.

Instead we have people wondering where they can click for regulatory compliance or if they can buy Compliance as a Service. [CaaS]

This is straight out of the NIST document. It’s what they’re working on. It’s worth knowing where your tax dollars are being spent.

..and I’m back with basics on what makes a good metric.

Metrics should be inexpensive. This means automated generation and gathering. This removes collection as a major source of errors and puts the “it magically happens” sense of wonder into the system.

Metrics should be interesting. If they’re not relevant, why did you bother collecting them?

Maverick metrics!

Verizon Business is pretty cool for releasing, not only the data, but a framework so that you too can release like data.

Awesome. Please everyone, do more things like this.

So you want to have a relevant metric program and not only show what empirically needs to be improved, but to show people why they should keep you around and continue paying you money?

Glad to hear it! It’ll be useful, I promise.

Some things are hard to measure, but people have found ways of finding indicators of symptoms anyway. A great example is of public health metrics.

Another tricky example is financial risk management because everyone finds money to be interesting. Those models are usually an entire talk in themselves and it’s been done many many times.

If you have tools, working reporting methods in your organization, and/or a framework to make use of, make things easier on yourself and use what you have available. Don’t make perfect the enemy of good.

Any tricks you can use to make information intuitive and digestible should be used.

Infographics are a good way to do it. Scorecards might work too.

Report only what is of interest and present solutions, not huge lists of problems. Keep the data that derived these interesting bits around in case someone wants more information. Use data to make your case for why you have come to these recommendations, conclusions, policy decisions, or staffing levels.

Data is the answer. It is the way.

So you want a metrics program but don’t know where to start? I have a couple of things for you to try.

First, think about what data sources you have around. Read this talk. Do you have application data or logfiles? What about a SIEM? Chances are you have loads and loads of data sources from which to glean metrics.

Ok. So how do you do it?

Look at some business intelligence software. There was one talked about at Metricon, but I suspect that I may like this one more. This may be just because they have a cool demo and can grab data from a variety of sources.

Don’t have a SIEM? Try playing around with the free license of Splunk.

Can’t figure any of this out? I used to work with a guy who started a company to help you out called Bitwork. They can give you metrics gleaned from your internal data and delivered in a SaaS model. Tell them what you need and let them figure it out for you.

Did you read this far? Cool!

I was a little unhappy with how it turned out as I thought that it was a bit vague and confusing, much like the current state of the industry, but I was told that many in attendance enjoyed it. Good enough.

Here’s some resources and additional reading.

CIO Mag: The Metrics Trap

Security Metrics: Replacing Fear, Uncertainty, and Doubt

Information Security Management Metrics: A Definitive Guide to Effective Security Monitoring and Measurement

securitymetrics.org and Metricon.

Truth to Power

A big thanks to the many people who were kind enough to discuss this topic with me for untold hours. I appreciate it!

The Trials of Toorcamp

Ian Gorrie — Thu, 16 Jul 2009 23:40:54 +0000

Toorcamp was many things this year.

It was fun.

It was uncomfortable.

Dustdevils ate things occasionally.

It was turbulent due to the trouble with Levitate to get hackers to help promote their event for free or they wouldn’t fulfill their agreement to let us use the missile facility for talks and workshops.

There was some excellent music.

There were fine people in attendance as it took some dedication and preparation to get out there and stay there.

Enough said about that. I was expecting more problems. More can be found at the Toorcamp wiki.

My presentation at this Toorcon Seattle area hacker retreat was concerned itself with three main points.

How to get a job in todays market
Identifying the common players and bad actors in todays organizations
How I recommend dealing with them

I entitled my talk Hacking HR in the traditional usage of the word hack. I’ve seen a lot of usage that uses “hack” as a synonym for small tips on how to accomplish obvious tasks. This isn’t how I use the word.

Anyway, let’s get started.

There are some really large problems with our industry at the moment, and they’re not improving. Things are getting worse. They’re getting more complex. There are people who don’t understand their complex systems taking bad advice from people who have profit motives that are not aligned to their customers best interest.

This is a prescription for bad times and, in general, they are upon us.

Bad times are not without opportunity and there is no time like the present to get started.

This is a talk in three acts (without trying to be overly pompous, just for pacing really)

How to get the gig you want against all odds
Identify the common players and bad actors in organizations and my suggestions on how to deal with them
How to effectively change the playing field. Fight bad actors with metrics and data. Change behavior by re-aligning profit motives.

The current state of the industry is in pretty bad shape.

Compliance drives and funds most IT and security efforts which results in ineffective and cart before the horse risk management and security governance programs.
More contractors are empowered and employed instead of FTEs
Hiring managers less relevant in todays hiring and management process.

These are huge problems.

Get powerfully clued individuals out of contracting/consulting/specialty and into positions where they can make an effective difference.

Empower insiders to make effective change and turn around industry trends of ineffective speciality, ineffective governance, and ineffective outsourcing. No consultant can be as effective as an insider who is deeply familiar with the environment, business units, and corporate culture.

The proper mindset. Levity included. There should always be an element of fun in these talks.

A quick blurb about me.

Companies that recognize the rewards of a good risk management program, like insurance and sometimes financial organizations, trend better. Usually for everyone else, important matters need to be presented in terms of business risk or opportunity that everyone can understand. Engaging in this discussion is one of the most important and rarely effectively performed tasks for those in our line.

I consider the things that I describe in my talk to be common and pervasive in moderate to large sized businesses in the United States and in places that follow American business’s lead.

My conclusions are based on my career of consulting and long conversations along this line with many of my trusted peers. I’m confident that you’ll see things my way. If you do not and disagree with me, I want to hear from you.

Act I: Breaking the ice and getting hired.

The current state needs to be understood. If you’ve ever wondered why some people who are not only not good at their role, but really obviously bad at it reached their position, this might help.

How are these bad actors allowed to get into organizations and reap huge rewards from not working towards their employers best interests? It’s because and in large respect, doing the right thing is not what gets rewarded.

The real problem: It is now commonplace that few understand how to effectively manage or hire anymore. The ninjas have been promoted up and away or running their own businesses and the losers have been fired or promoted just enough to make sure everyone else fails. Conflicts of interest are rampant with vendors and are in opposition to their clients running healthy risk management programs.

No one in senior management roles seem to have any clue about technology and treat it as a luxury instead of the bedrock on which modern business is performed.

The right people aren’t rewarded, the right skill sets are not valued and cultivated, and organizations cant attract or retain the right people and skill/experience sets they need to run an effective information security program.

Disclosing these methods and interests to the internet in general will, I hope, change the way business is done over time.

The first step is getting in the door, so how do you get an edge on that position that you want?

Look them up on social networks. Stalk them and cordially meet them at user groups and professional organizations if you’re really motivated.

Use LinkedIn to get insider contacts and internal intel for the players and the organization you’re trying to enter.

Use search engines and social network mining for greater impact. Don’t be shy.

wink.com – Searches on people over social networking sites.

pipl.com – Basically a people-optimized search engine. It’ll help narrow down likely results of interest.

Image credit

Dress right. Not overdressed. Not underdressed. Example: geeks in suits freak out hiring managers whos “dress up” is cleanest-t-shirt and jeans with least holes.

Be a right-fit. Remove overly qualified statements, degrees, or certifications from your resume. Just because you can, doesn’t mean that you should volunteer information that might make you sound bragging or overqualitied. Understated is a good tactic. Be surprising.

Get contact information for those you interview. Consider thanking them for their time and for meeting them. This isn’t always a good idea, but is a class move if the audience is receptive.

Try not to give up any dealkillers. Don’t be late. No one cares if there was a traffic accident on the highway. Don’t have dirty fingernails. Hiring managers have odd dealbreakers sometimes. Try to avoid the common ones.

Staffing is about liking you. Jerks can get gigs occasionally, but only if there isn’t a guy who isn’t almost as good that people would like to work with more.

If you’re going to be an ass in business, you had better have all the answers all of the time to make up for it. It’s usually a better idea not to be a jerk. It’ll make you a stand out; a nail to be hammered.

Be known in the community offline and on. Give back. Write things. Contribute. All of these things help.

It would be better to do useful things, but I’ll bet you can think of some examples of people who have become big deals just for talking to people and being knowledgeable.

Ever submit your resume for a position you were qualified for but never heard back from anyone? It might be because someone is screening applications and looking for keyword matches. It happens all the time. It’s a lousy fit for technology positions, but no one told the human resources industry.

But don’t overdo it. Tailor it to the opening writeup.

If you don’t know someone specific, don’t use a cover letter.

Have a well formatted and presented resume. A bad resume is almost always a dealkiller.

Follow up. Be enthusiastic.

So what’s the problem here? Why doesn’t merit rise to the top and why don’t poor performers get culled from the herd?

The simple reason is that when people get together, things get complicated.

The recent compliance efforts have not got the job done. Worse, most people and many in the industry, don’t know the difference between effective governance (to use an overused and frequently misused term) and just making the minimum effort which is compliance.

A lot of things were funded because of the big scary compliance boogyman, but in general it has only created a huge mess of policy, standards, procedures, outsourcing, controls, contracts, vendors, complicated staffing and dependent org charts, and more.

Sounds complicated? It is.

..and guess who that is going to be.

It’s the attendees of this talk and those like us that are going to be tasked with the big cleanup after conventional wisdom comes back around to reality that convenient and magic bullet solutions aren’t working. It is not going to be pretty.

It’s going to take a lot of work. Things are going to change. Empires are going to fall. Castles built of shifting sand are going to fall into the sea. The current common model is not sustainable and isn’t doing anything for shareholder value. Eventually that will be what brings things around.

The first step is becoming an insider.

Deliverables from important consulting engagements can be left unread. Without commitment from executives or a board, a risk management or infosec program is toothless and can be ignored.

Insiders have a level of familiarity with business practices and behaviors that consultants and contracting outsiders do not by their nature of short-timers. This needs to be valued and leveraged more than it is currently.

Risk to technology systems often isn’t considered a business risk, but a cost center. The benefits are usually overlooked and not capitalized upon.

The root cause here is corporate culture.

There’s a million references out there about why being an agile organization is a good idea. I can only guess at why executive leadership doesn’t make it a bigger priority. The model seems to be worth talking about, but it seems to be rare indeed that anyone wants to take on a difficult job pre-catastrophe.

Two generalized corporate culture examples:

American: ready shoot aim aim aim
Japanese: ready aim aim aim aim shoot

Selecting a solution to poorly or undefined problems and fixing deployments of poor-fit solutions can be very hard. If they can’t be fixed, they will be very expensive to operationally support.

Metrics bring an appreciation of quality and total cost. Both are lacking. What is needed, and what are usually unavailable, are more apples to apples comparison of risk and reward. Cherry picking of statistics for TCO and ROI calculations is rampant.

The RFP process: The low bid is often sizably more expensive than others when total ownership and operational cost is considered. Efficiency and elegance has hidden rewards.

Risk management and even assessments are not quantitative product. They are qualitative art.

Specialists, in my experience, tend to have linear and routine thinking in bringing the same approach to every problem. This can yield incomplete answers and piecemeal solutions to complicated problems. Piecemeal means complication, fault intolerance, and expensive operational cost.

Separation or segregation of duties is a good idea and is appropriate often, but that doesn’t mean that there should be a limited awareness of processes and overall architecture.

Reasonable people are often scared off from the technology industry. There are a lot of reasons for this and that could be an entirely different talk.

These people are not a signifigant part of the problem as they can be reasoned with effectively. They’re out there and I hope you can find them.

There are often many solid contributers in successful organizations.

Many of the people I’m about to mention can be effective. I’m going to center on what they’re like when they’re a problem and my take on how to interact with them.

Put things in terms relevant to their interests.

Look out for their ego measuring contests. Outshining them is a sure way to land in their disfavor.

These guys are pretty much irrelevant but common. They are best avoided.

They’re going through a checklist. Give them things to check off and move on to the hiring manager(s).

They want you to sound confident. Very likely to have no idea what you do, why it is important, or how it gets done. They’re looking for you to speak well and sound like you know what you’re talking about.

Sadly too common with downsized efforts, eroded budgets, and no resources to which they can delegate.

They’re looking for someone who can work in a vacuum. Sufficiency is what you need here.

If you’re fortunate enough to interview with a ninja, make the most of it.

BS the ninja at your peril.

Come clean. Tell them what you know and what you do not.

Tell stories from the trenches.

So if we can’t count on insiders to get everything done because the old clue has been promoted or driven out, who’s left?

The previous slides are all mostly hard workers in their own way. The following are not.

Every consultant has worked with this joker.

They can create big problems and large messes of rushed or under-delivered projects that actually have to work.

After signoff, under-baked solutions can be a real operational nightmare. The more complicated and ambitious, the worse the aftermath can be if anything other than ideal.

Can be even less interested in the outcome than all promises sales guy. Relentless in their “buy our stuff. we’re the best” mantra.

When HR doesn’t want to do their job and hiring managers are too busy or not allowed to be involved, the staffing firms soak up a lot of profit by just posting requirements and funneling in bozos.

Getting into a relationship with HR means they can sit at home and capture a significant portion of contractor work effort compensation while adding little (if any) value.

If that wasn’t enough, they also have a profit motive to place as many people as possible, not in placing quality people. Individual headhunters can achieve in extra ordinary ways, but staffing firms almost never deliver in this way.

These people are key actors in the not-my-job industry of lazy.

More times than not, their goals are misaligned to the organization that is employing them. Very rarely is a consultancy interested in solving problems. There’s no profit motive. They’re interested in an increase in revenue and scope of engagements.

The larger the size of the consultancy, the more likely these practices are to arise.

The real magic of the magic quadrant is the ability to get people to pay for the analysis.

Scoped to the average environment in the average business in the average industry.

There is no industry average environment. The best fit for an environment may not be on the leading edge of their wave or quadrant at all.

Yes. You really do have to do your homework.

The classical argument of heterogeneous and homogeneous networks and solutions is usually academic, however interoperability and performance is often misstated or exaggerated.

This individual is the complement to the Industry Analyst. Neither is a replacement for skilled investigation and logical comparison of options.

What is more expensive? A failed implementation following a hasty decision, or a reasoned approach?

My take on how to dig our way out.. but first!

Compliance is a minimum standard, not a gold standard. It is a checklist.

It is not a risk management program or effective governance.

An auditor background and skill set and that of an information security practitioner very rarely intermix.

Harder, Better, Faster, Stronger. This is the way. Always be improving.

Dead Mike knows what was up.

Source. CB4 Video.

Edward Tufte also knew what was up. I’m told that his works are amazing.

Metrics are most effective when cheap to collect and immediately meaningful to the reviewer.

This is a difficult but highly rewarding standard to achieve.

These were some metric suggestions in order to inspire discussion and interaction during my talk.

Some people were pretty heated.

Some didn’t believe that scoring candidates was feasible. It was my contention that academic boards had found effective ways to do just that with their incoming student applicants and surely simple quantitative metric data can be gathered.

One attendee mentioned counting spelling mistakes in a resume.

Another suggested that any metric collection can be gamed nearly immediately. I suggested not disclosing the metric criteria.

If you have your ducks in a row, it will call attention to those that do not. If this does not happen, call attention to it.

Tell the world! Share your data!

When you can rely on data, you can make effective decisions in the light of day based on something more than arbitrary judgement and gut feelings.

When this is pervasive, FUD will be a thing of the past.

Consultants at large failing business are delaying the inevitable unless culture change takes place. The axe man will appear one way or another.

Metrics are factual. They are not slander.

“Oh! Someone might sue you!” That’s what corporate retained counsel is there for. Sharing data in a pay it forward fashion will make the business community and our industry a much better place nearly overnight.

This is important. This needs to happen.

If you enjoyed this talk, you may wish to look at one of my previous talks about security and compliance metrics (a long talk) or the added risks of compliance (a short talk).

Hopefully my sense of humor comes through in this publication method. I attempt to present on issues that I have not heard aired previously in a light-hearted and whimsical way, and only when I feel I can contribute something to the conversation.

Thanks for reading. I’d love to hear from you.

My talk at Seattle Toorcon 2008

Ian Gorrie — Sun, 20 Apr 2008 05:52:38 +0000

I gave a little talk this weekend at the second Seattle Toorcon.

My presentation is as follows, though as usual, I ad lib when presenting. Video may appear in the future.

The compliance game: The enemy of good

Lots of execs have the idea that technology is a cost center and not the bedrock that enables their business to function.

This leads to reckless activities caused by not treating risks to their information systems as they would other business risks, (and also because of what has become the usual reactions to fraud and appropriate disclosure to investors getting punked)

So, with Sarbanes-Oxley and others, now if you’re an exec and you aren’t doing the job you were hired to do,

they can put you in jail when it all hits the fan.

Wait! I’m an executive! Jail is bad! I don’t want to go to the rape camp!

What should I do?!?

Typically, you can overreact and, instead of doing what you should have been doing in the first place, you can do something that is obviously better; you can dump as much money as you can find at the perceived problem of making sure that your surpass the standard of due care in your industry to be “above average.”

Bring in the consultants! You need to be better than average else you might be going to camp. Since everyone has to be better than average, costs and efforts increase and increase.

This is the same reason that executive compensation is 100s of times greater than the average employee in America.

[ Someone should come up with a better behavioral term for this. ]

So, in much the same way executive compensation is on geometric curve, compliance standards follow.

So are you safe now?

Does this fix problem? Yes!

Well. Kinda… or maybe not at all.

Maybe even worse than before you spent all that money

This will likely give great improvements to those that are way behind, but it can also defeat it’s own efforts.

One of my favorite examples of compliance gone wild is password enforcement:

Since passwords are such a foolproof way to police complicated systems and responsibilities, deploying a system to strengthen authentication isn’t what you should do. You should really just change passwords a lot.

Oh. They should also be increasingly complicated so that no average worker will remember them. You should also make them change it every week or two on a ton of systems so that your workers spend a lot of time changing and forgetting their passwords…

unless they start writing lists.

But we tell them not to do that! Guess what. Everyone does it. If it’s not in a hard copy hidden under their keyboard or a collection of post-its, then they are cached on their workstation somewhere… or a bunch of enable passwords in their wallet. I’m sure you can find an example of this in the next office of a public company you’re hanging around.

Another great one is segregation of duties. It’s the idea that every role’s responsibility should be paired with another role that will catch them if they’re being shady and vice versa. It’s foolproof! What an awesome plan!

Where it may be the case that it is somewhat effective in prevention or commoditization of their workers, what is assured is that in complex technical environments, no one person or team will be equipped to deal with the interdependent systematic problems. Unfortunately, those tend to be the really critical ones.

Segregation of duties for audit and risk frameworks when too zealously applied mean that skills become specialized and no individual is allowed to have a complete understanding of operations. If no one retained on staff has a effective holistic understanding of complicated systems, solutions can become piecemeal and unreliable. Staff retention becomes a larger problem as tasks become more repetitive and narrow.

You can always try mind control.

In summary and in short, nothing fixes companies that are doing it wrong. This is because the deterrent of fines is treated as a cost of doing business and the idea of public shaming of bad behavior seems not to be effective. We are left to choose between the threat of jail and fines. Jail is too much of a motivator and leads to over-reaction, and overblown controls which can be (and usually are) counter-productive to what is good. Fines can be ignored as a cost of doing business. Their efforts to be “perfectly compliant” can become the enemy of good business and efficient environments. Look for these behaviors in the future, and attempt to resist more controls to counter the controls that they are there to control.

..or alternatively for this audience, become familiar with their practices and work to exploit their many weaknesses.

ITCi 2007

Ian Gorrie — Mon, 12 Nov 2007 22:42:41 +0000

This is the presentation that I gave earlier this week at the ITCi Conference in San Diego, California. It was well received and fostered a lot of interesting discussion.

My recording of the event on my laptop had enough problems as to be distracting, so I gave up on using it to export a real-time presentation. Instead I will try to give my speaking points inline with my individual slides. If a good audio recording becomes available, I will kick out a video format of this presentation synched with discussion audio. I was hoping to make use of some of the new Keynote functionality, but the audio and speaking position setup was a little questionable and I was unable to see my speaking notes, so I winged it freestyle. Everything seems to go well in a free form way.

Anyway. On to my presentation.

Everyone loves a title slide.

Fifteen seconds about who I am. I wanted to make it clear that I am a technologist and can actually discuss solutions to these problems as I am a huge geek; I live, eat, and breathe this stuff.

A couple of good quotes to get the audience into the mindset I’m going for here.

An overview of how I’m going to address the topic at hand.

Speaking points:

Anything that is measured in “Low, Medium, and High” should be included with “bad metrics.”
If metrics are too challenging to understand, they will not be readily business relevant.

A great graphic from one of the definitive works in this subject area which I make frequent reference in this presentation:

“Security Metrics: Replacing Fear, Uncertainty, and Doubt” (Andrew Jaquith)

Though I disagreed with some of his advice toward the latter half of the book, I found him right on through much of it. I gave away a copy of this book at the end of the discussion to the person in the audience that contributed the most. I let the audience decide who that was, and it was pretty fun.
Speaking points:

Mention: Continuous Audit & Risk Assessments – Paul Reymann, Norbert Kuiper (a previous talk at the conference)
The hamster wheel methodology of periodic identification, freak out, remediation and new tool identification lacks valuation and prioritization and, Andrew Jaquith suggests, is only the easy part of risk management.
Symptomatic problems, not systematic. Root causes remains elusive.

Speaking points:

Assets on servers, workstations and mobile devices? In aggregate?
So really what you want from metrics and data is making your organization versatile, flexible and quick to adjust to change and adversity.
I think of metrics as being like vectors; not only does it need to have a value, but it needs to show direction.
Through measuring, your organization will be able to react quicker to change. You will know if your projects/controls/implementations are successful, how much they cost in terms of real dollars, and be able to track operational efficiency.
What use is deploying these costly frameworks, technical implementations, and policies if you can not track their effectiveness and make effective changes to improve them?

I pick on the vagueness of ‘threat’ metrics.

So I covered what the bad data problem is all about. What do we want instead?

What is key here is to identify the vital essence of your organization. What is key to the success of your business?There should be at least one and not more than a few key metrics.

Also worth thinking about are key metrics for your business unit or department. What shows your effectiveness and success most effectively?

Granularity of data, which is collected in greater amounts than any company I have been able to find, allows Amazon to find errors based on behavior of this metric.

If traffic spikes, someone may have listed an ipod for $5 and word is getting around. If traffic drops off, there may be an outage or a performance hit somewhere.

Activity outside the standard delta or standard deviation can be quickly detected and analyzed to the benefit of the agile organization.

The aggregate score of vulnerability scanning of their production network. Focus on monthly delta to determine handling of risk and effort allowances.This was sufficient for his board in measurement of exposure and change in risk in their critical environment.

Amazon’s SOX and PCI compliance requirements followed their financial systems. Other systems were out of scope, not business critical, and therefore not in scope for compliance. Compliance, since they already had lots of evidence and effective diligence in these matters, was a fairly simple matter.

Speaking points (some optional notes seen in my post-it style note in the slide which would not be visible to the audience):

ISO 27004, but it will very likely be more of the same that is already available in NIST SP 800-55
Frameworks offer no practical recommendations on managing or monitoring and are highly open to interpretation. That would be why we are here at this conference and compliance is a billion dollar industry full of hand-waving
ALE may show that valuations of A > B, but that’s about it. Long rant found about ALE found in Security Metrics book. [Single Loss Expectancy, Annual Rate of Occurrence]

Not a huge fan of this template. I noticed that hardly any of the other presenters used it.

I went through this slowly step by step to illustrate how this is a conventional, unclear, and possibly meaningless process.

Speaking points:

This is popular in government. As seen on c-span. Is measurable, but it can be unclear what exactly it is measuring.
The Assessment vs Audit is a perennial topic with me. The different goals are the important differentiation.

Speaking points:

Security risks are especially variable
What unified platform is available? This seems to be where most talks leave off. But not today.

Dun dun dunnn.

This is the centralized platform strategy I had alluded to many times in my talk previous to getting to this point.

Data sources are usually ready to integrate out of the box assuming application uses standard conventions
I would recommend having at least one metric to record and track the progress of every major deployment. Automated generation should make this a minimal cost and your organization will be able to track it and be able to prove its success or correct its failure in real time instead of waiting for the next self assessment, audit, or tangential operational indicator (where many organizations with lacking systems actually detect anomalous behaviors; when they impact production systems because of capacity or instability)

Speaking points:

Increased cost may be in the cost of managing multiple platforms for the same data generating tasks
Lacking enterprise vision can be fine, but not if there is the possibility of duplication of effort yielding inconsistent results. This validates the benefits of a single architecture.
Though SIM solutions tend to already have a very robust offering of that functionality

Forrester has some interesting market projections here speaking about how the growth of the SIM market will start to capture the attention of moderately-sized businesses. I think that, if this happens, it will be because of the smaller and more affordable options in this market that do not require a capital investment in appliances or dedicated infrastructure.

My presentation selecting an enterprise SIM is available here.

If SIM deployments become the industry standard and you do not have a system that performs as well deployed, your organization may be at risk of appearing not to be in keeping with industry norms if an event occurs. Your legal council, and possibly compliance teams, should be on point in this vague (to me) area.

The Jaquith book pushes the Balanced Scorecard pretty hard, which is sound advice. My point of disagreement is that I do not like to advise clients, or anyone, to attempt to revolutionize all behaviors in one fell swoop. Additional reporting frameworks are risky to implement, because people (and therefore organizations) already have ways they feel comfortable doing things.

SIM reporting can be incorporated into any existing reporting structure and, through a series of mockups and pilot reporting methods, you can warm your executives into desiring this information instead of ramming it down their throats by c-level mandate.

In much the same way I advocate using whatever compliance framework is the best fit for the organization, instead of whatever framework your advisor is the biggest fan of, in order to have the least risk of adoption and easiest transition into routine use.

I should have covered all of these points repeatedly in the discussion, but it’s always good to point out key points that I’m endeavoring to express again at close.

Empowering insiders to surmount these challenges and goals are the best way to have them conclude in a successful result.

This may be an unpopular opinion, but I believe it to be an important one. I’ve seen too many resources applied to these challenges without sufficient leadership and internal knowledge in the past. It leads only to ineffective situations, an inefficient workflow, and a large bill.

Here I made time for anything more to discuss. Like the location of this presentation, for instance. (Here it is!)

Some of the references mentioned without which this presentation would have been considerably more difficult to put together.

ISO 27000 series. Look for 27004 coming Real Soon Now.
NIST SP 800-55 is a good primer, but not gospel. Look to Appendix A to wet your appetite for security metrics.
I’ve mentioned Security Metrics by Mr. Jaquith a few times already, but I’ll do it here again.
Forrester Research for market analysis

..and thanks for coming. Please feel free to contact me regarding any lingering questions or advice. I’m happy to help.