Here is the abstract from Unskilled and Unaware of It: How Difficulties in Recognizing One’s Own Incompetence Lead to Inflated Self-Assessments (Justin Kruger and David Dunning, Department of Psychology, [...]]]> I wanted to continue a bit where I left off with a non-technical explanation of what people such as myself do and my commentary on evolving technology management.

Here is the abstract from Unskilled and Unaware of It: How Difficulties in Recognizing One’s Own Incompetence Lead to Inflated Self-Assessments (Justin Kruger and David Dunning, Department of Psychology, Cornell University), a fairly well known publication that appeared in the Journal of Personality and Social Psychology (official link unavailable):

People tend to hold overly favorable views of their abilities in many social and intellectual domains. The authors suggest that this overestimation occurs, in part, because people who are unskilled in these domains suffer a dual burden: Not only do these people reach erroneous conclusions and make unfortunate choices, but their incompetence robs them of the metacognitive ability to realize it. Across 4 studies, the authors found that participants scoring in the bottom quartile on tests of humor, grammar, and logic grossly overestimated their test performance and ability. Although their test scores put them in the 12th percentile, they estimated themselves to be in the 62nd. Several analyses linked this miscalibration to deficits in metacognitive skill, or the capacity to distinguish accuracy from error. Paradoxically, improving the skills of participants, and thus increasing their metacognitive competence, helped them recognize the limitations of their abilities.

This principal, known now as the Dunning–Kruger effect, was given the dubious honor of winning an Ig Nobel prize in 2000. Astonishingly enough, media from the NPR Science Friday show that covered the Ig Nobels that year is still available 10 years later.

It seems to be part of the human condition to have trouble recognizing both competence and incompetence in matters where the observer is not at the top of their game. For example, nearly all Americans and most Swedes think that they are better than average drivers.

This is entertaining and all, but I say this not to call attention to making fun of stupid or incompetent people in the world. Rather, how does one identify and work with people who are incompetent and not cognizant of it? What about if they are highly intelligent and are undervaluing their abilities? How does one know if ones self assessment is accurate? How can one right-size ability and decision making stature? How can one make a qualitative judgement of qualitative judgements?

It’s not a simple problem.

Certainly being a well rounded and traveled individual may help in finding this kind of clarity. This may also be something largely gained from journeyman tradecraft; seeing other methods and masters firsthand. This may be why people talk about risk management as being on par with a JD or MD; it takes a lot of time, passion, and diligence to become and stay competent and aware and literate of the many challenges present in diverse environments and constantly moving technology. The responsibility in the design, management and assessment of complicated systems is also large. Persistent errors here can literally cost lives, crash fortunes, and wreck business models. It may be one of the hardest careers.

The massive efficiency increases to work and leisure of the last 20 years, the strides that have been taken in the knowledge of how individuals most effectively learn, and with all of the information available on the internet, affords the opportunity to learn effectively and in a disciplined approach to become and stay competent in this field. I find that many people make casual reference to Gladwell’s “~10,000 hours = success in a field” rule which seems to be the take-away factoid from his book Outliers. I’m not really sure what to think of Gladwell and his writing, but it strikes me arbitrary. I think of competency in a large and deep field of knowledge to be like snowballs rolling down a hill; they start small, but they increase in size, depth, and have a sense of momentum.

Everyone can’t be a world famous polymath like Leonardo da Vinci or Johann Wolfgang von Goethe. Genius of that level seems to be cultivated only occasionally and is not a realistic role model for nearly anyone. I do know a lot of people that are like Richard Feynman however; a pretty competent man and likely fun at parties.

“A human being should be able to change a diaper, plan an invasion, butcher a hog, conn a ship, design a building, write a sonnet, balance accounts, build a wall, set a bone, comfort the dying, take orders, give orders, cooperate, act alone, solve equations, analyze a new problem, pitch manure, program a computer, cook a tasty meal, fight efficiently, die gallantly. Specialization is for insects.” — Robert Heinlein

I do believe that many of this ages competent [wo]men are involved in information technology and the cream of IT are those that manage complicated systems. Certainly many of the geek cultural heros are, such as the Batman, Gregory House, Ryo Saeba, and James Bolivar DiGriz, show the common geeks aspiration to be widely competent. This may be the true root of what geek culture is all about.

Predictably to those familiar with my past work, this leads back to my usual harping on metrics and data analysis instead of so called best practices, third party industry rankings, or arbitrary standards. Meaningful data is the only way to get away from the constructs of cognitive bias that lead people to fear plane crashes and terrorism but feel completely safe driving a car in nearly any condition.

I pose a question: can the competence of a complicated system be left to a specialist who is not competent to judge the quality of all of its components? The stakes are high after all, for what could be more important in a world that runs on the power of information but the secrets of powerful people and organizations? I contend that a conformist drive to make industry into a uniform, standard, and specialist product is directly at odds with that of producing quality work product for our industry as a whole. The business would prefer that an employee can be easily recruited, fired, or trained to perform this job, but I do not believe that this is a realistic goal for risk management specifically, and perhaps, IT in general.

In reading one of Andre’s dives into the waters of competence and speciality, I was stuck at his harsh but close to home hitting words about the state of the industry and professional credentials. He invokes Dan Geer. Here is the preface to the talk he cites, a keynote at Source Boston:

Good morning. If you were to come to my office, you would see on the wall these four rules:

• Work like Hell
• Share all you know
• Abide by your handshake
• Have fun

And later:

Only people in this room will understand what I am now going to say. It is this: Security is perhaps the most difficult intellectual profession on the planet. The core knowledge base has reached the point where new recruits can no longer hope to be competent generalists, serial specialization is the only broad option available to them.

Ah there it is again. Serial specialization might be closer to what I mean, but it is a bit hairsplitting to argue about if a skill set is generalist or serial specialist. I think the real criteria here is passion for learning and a will to do what is necessary to learn and work well.

I was discussing this with a friend who is a developer who has been kept pure from this circus but hears some of my tales. He had this to say when I asked him about his experience with efforts to make a homogenized software development environment. He asked that he not be named but his words from the developer side of the house sounded quite familiar to me:

Having worked in a variety of industries, at a number of companies, provides one with a reasonable substrate against which to compare and contrast the natures of the environments in which one works. A set of patterns emerge that reveal the nature of an organization. Unfortunately, many of them reveal weaknesses or inefficiencies in the modern place of employment.

College degrees are all but an afterthought nowadays. Employers have responded by creating a set of artificial restrictions. Since I started at my current employer, a policy has been enacted by the HR department, stipulating that an applicant seeking, for example, a software development position, must hold a degree in computer science. This is a terrible idea. What does HR know about finding talented programmers? Perhaps the most talented programmers I’ve worked with have been the ones that do not hold a computer science degree. A couple of them never even finished college. An ability to excel in software development is not restricted to those who have spent years and a bunch of money on formal training in the field. As with most other crafts, a good indicator of success is the zeal with which one pursues it. Why not allow those who are responsible for developing the software determine the qualifications for those who will ultimately be hired to help build it? Senseless policies like these disempower those accountable for the work and serve as a blockade to workplace efficiency. It’s not much of a stretch to feel like HR, in situations like these, is a congressional body for the workplace, passing more and more laws as though it’s an effort to convince others that their existence is justified.

Something else that becomes clear after working in various employment climates is how important it is to hire really good people. This sounds like an obvious statement, but chances are you don’t even know how much this affects you. Hiring great people is what allows a company to trust employees to be able to do their jobs with minimal intervention. Being able to trust employees in this way reduces the need for artificial barriers that stand in the way of getting real work done. These policies are often in place to protect a company or workgroup from someone doing something stupid. Employers should aspire to hire good people–the best they can–across the board, so that they don’t have to annoy the good people with policies or artificial divisions of work responsibilities that mainly get in their way. This is, of course, easier said than done, but if this approach were more prominently represented in hiring practices, I suspect we’d all be better off. But, hmm, where would everyone else work?

Practically speaking, it seems that the companies that are known for difficult interviews are the ones that have a higher caliber of employee across the board. I’m not talking about making interviews hard and annoying for its own sake, but rather interviewing “smart,” and determining a good fit, not necessarily who has paid the right amount of money for professional training.

Additionally I would like to mention that with the great responsibility and knowledge required to perform well in these critical roles, there is the sticky point of ethics and professional due care. The current industry standard of “not getting caught being a jerk” is pretty weak. This relates to my point of competence and generalist/specialist bias as there is a classical example that could be considered as a useful metaphor: Bushido. Will has his own rant on the subject which I recommend.

In the end, the world is wide and filled with wonder and things that we as humans will never fully understand. I would suggest, as an ideal, not to fear the unknown but appreciate it for the challenge that it is to do something that you haven’t done before. There will always be people who have something to teach you. Learn from them.

At the end of the day, it is the seasoned veterans from the trench warfare of operations and gladiatorial arenas of the boardrooms that will steer a ship to calm safe waters. They have the best stories, have seen the mountaintops, have looked into the yawning abyss, and have come to you to tell the tale. Are you ready to hear them?

Image References:
Creating Passionate Users
Dave Gray

Related posts:

1. What we do

It was fun.

It was uncomfortable.

Dustdevils ate things occasionally.

It was turbulent due to the trouble with Levitate to get hackers to help promote their event for free or they wouldn’t fulfill their agreement to let us use the missile facility for talks and workshops.

There was some excellent music.

There were fine people in [...]]]> Toorcamp was many things this year.

It was fun.

It was uncomfortable.

Dustdevils ate things occasionally.

It was turbulent due to the trouble with Levitate to get hackers to help promote their event for free or they wouldn’t fulfill their agreement to let us use the missile facility for talks and workshops.

There was some excellent music.

There were fine people in attendance as it took some dedication and preparation to get out there and stay there.

Enough said about that. I was expecting more problems. More can be found at the Toorcamp wiki.

My presentation at this Toorcon Seattle area hacker retreat was concerned itself with three main points.

1. How to get a job in todays market
2. Identifying the common players and bad actors in todays organizations
3. How I recommend dealing with them

I entitled my talk Hacking HR in the traditional usage of the word hack. I’ve seen a lot of usage that uses “hack” as a synonym for small tips on how to accomplish obvious tasks. This isn’t how I use the word.

Anyway, let’s get started.

There are some really large problems with our industry at the moment, and they’re not improving. Things are getting worse. They’re getting more complex. There are people who don’t understand their complex systems taking bad advice from people who have profit motives that are not aligned to their customers best interest.

This is a prescription for bad times and, in general, they are upon us.

Bad times are not without opportunity and there is no time like the present to get started.

This is a talk in three acts (without trying to be overly pompous, just for pacing really)

1. How to get the gig you want against all odds
2. Identify the common players and bad actors in organizations and my suggestions on how to deal with them
3. How to effectively change the playing field. Fight bad actors with metrics and data. Change behavior by re-aligning profit motives.

The current state of the industry is in pretty bad shape.

• Compliance drives and funds most IT and security efforts which results in ineffective and cart before the horse risk management and security governance programs.
• More contractors are empowered and employed instead of FTEs
• Hiring managers less relevant in todays hiring and management process.

These are huge problems.

Get powerfully clued individuals out of contracting/consulting/specialty and into positions where they can make an effective difference.

Empower insiders to make effective change and turn around industry trends of ineffective speciality, ineffective governance, and ineffective outsourcing. No consultant can be as effective as an insider who is deeply familiar with the environment, business units, and corporate culture.

The proper mindset. Levity included. There should always be an element of fun in these talks.

A quick blurb about me.

Companies that recognize the rewards of a good risk management program, like insurance and sometimes financial organizations, trend better. Usually for everyone else, important matters need to be presented in terms of business risk or opportunity that everyone can understand. Engaging in this discussion is one of the most important and rarely effectively performed tasks for those in our line.

I consider the things that I describe in my talk to be common and pervasive in moderate to large sized businesses in the United States and in places that follow American business’s lead.

My conclusions are based on my career of consulting and long conversations along this line with many of my trusted peers. I’m confident that you’ll see things my way. If you do not and disagree with me, I want to hear from you.

Act I: Breaking the ice and getting hired.

The current state needs to be understood. If you’ve ever wondered why some people who are not only not good at their role, but really obviously bad at it reached their position, this might help.

How are these bad actors allowed to get into organizations and reap huge rewards from not working towards their employers best interests? It’s because and in large respect, doing the right thing is not what gets rewarded.

The real problem: It is now commonplace that few understand how to effectively manage or hire anymore. The ninjas have been promoted up and away or running their own businesses and the losers have been fired or promoted just enough to make sure everyone else fails. Conflicts of interest are rampant with vendors and are in opposition to their clients running healthy risk management programs.

No one in senior management roles seem to have any clue about technology and treat it as a luxury instead of the bedrock on which modern business is performed.

The right people aren’t rewarded, the right skill sets are not valued and cultivated, and organizations cant attract or retain the right people and skill/experience sets they need to run an effective information security program.

Disclosing these methods and interests to the internet in general will, I hope, change the way business is done over time.

The first step is getting in the door, so how do you get an edge on that position that you want?

Look them up on social networks. Stalk them and cordially meet them at user groups and professional organizations if you’re really motivated.

Use LinkedIn to get insider contacts and internal intel for the players and the organization you’re trying to enter.

Use search engines and social network mining for greater impact. Don’t be shy.

wink.com – Searches on people over social networking sites.

pipl.com – Basically a people-optimized search engine. It’ll help narrow down likely results of interest.

Image credit

Dress right. Not overdressed. Not underdressed. Example: geeks in suits freak out hiring managers whos “dress up” is cleanest-t-shirt and jeans with least holes.

Be a right-fit. Remove overly qualified statements, degrees, or certifications from your resume. Just because you can, doesn’t mean that you should volunteer information that might make you sound bragging or overqualitied. Understated is a good tactic. Be surprising.

Get contact information for those you interview. Consider thanking them for their time and for meeting them. This isn’t always a good idea, but is a class move if the audience is receptive.

Try not to give up any dealkillers. Don’t be late. No one cares if there was a traffic accident on the highway. Don’t have dirty fingernails. Hiring managers have odd dealbreakers sometimes. Try to avoid the common ones.

Staffing is about liking you. Jerks can get gigs occasionally, but only if there isn’t a guy who isn’t almost as good that people would like to work with more.

If you’re going to be an ass in business, you had better have all the answers all of the time to make up for it. It’s usually a better idea not to be a jerk. It’ll make you a stand out; a nail to be hammered.

Be known in the community offline and on. Give back. Write things. Contribute. All of these things help.

It would be better to do useful things, but I’ll bet you can think of some examples of people who have become big deals just for talking to people and being knowledgeable.

Ever submit your resume for a position you were qualified for but never heard back from anyone? It might be because someone is screening applications and looking for keyword matches. It happens all the time. It’s a lousy fit for technology positions, but no one told the human resources industry.

But don’t overdo it. Tailor it to the opening writeup.

If you don’t know someone specific, don’t use a cover letter.

Have a well formatted and presented resume. A bad resume is almost always a dealkiller.

Follow up. Be enthusiastic.

So what’s the problem here? Why doesn’t merit rise to the top and why don’t poor performers get culled from the herd?

The simple reason is that when people get together, things get complicated.

The recent compliance efforts have not got the job done. Worse, most people and many in the industry, don’t know the difference between effective governance (to use an overused and frequently misused term) and just making the minimum effort which is compliance.

A lot of things were funded because of the big scary compliance boogyman, but in general it has only created a huge mess of policy, standards, procedures, outsourcing, controls, contracts, vendors, complicated staffing and dependent org charts, and more.

Sounds complicated? It is.

..and guess who that is going to be.

It’s the attendees of this talk and those like us that are going to be tasked with the big cleanup after conventional wisdom comes back around to reality that convenient and magic bullet solutions aren’t working. It is not going to be pretty.

It’s going to take a lot of work. Things are going to change. Empires are going to fall. Castles built of shifting sand are going to fall into the sea. The current common model is not sustainable and isn’t doing anything for shareholder value. Eventually that will be what brings things around.

The first step is becoming an insider.

Deliverables from important consulting engagements can be left unread. Without commitment from executives or a board, a risk management or infosec program is toothless and can be ignored.

Insiders have a level of familiarity with business practices and behaviors that consultants and contracting outsiders do not by their nature of short-timers. This needs to be valued and leveraged more than it is currently.

Risk to technology systems often isn’t considered a business risk, but a cost center. The benefits are usually overlooked and not capitalized upon.

The root cause here is corporate culture.

There’s a million references out there about why being an agile organization is a good idea. I can only guess at why executive leadership doesn’t make it a bigger priority. The model seems to be worth talking about, but it seems to be rare indeed that anyone wants to take on a difficult job pre-catastrophe.

Two generalized corporate culture examples:

• American: ready shoot aim aim aim
• Japanese: ready aim aim aim aim shoot

Selecting a solution to poorly or undefined problems and fixing deployments of poor-fit solutions can be very hard. If they can’t be fixed, they will be very expensive to operationally support.

Metrics bring an appreciation of quality and total cost. Both are lacking. What is needed, and what are usually unavailable, are more apples to apples comparison of risk and reward. Cherry picking of statistics for TCO and ROI calculations is rampant.

The RFP process: The low bid is often sizably more expensive than others when total ownership and operational cost is considered. Efficiency and elegance has hidden rewards.

Risk management and even assessments are not quantitative product. They are qualitative art.

Specialists, in my experience, tend to have linear and routine thinking in bringing the same approach to every problem. This can yield incomplete answers and piecemeal solutions to complicated problems. Piecemeal means complication, fault intolerance, and expensive operational cost.

Separation or segregation of duties is a good idea and is appropriate often, but that doesn’t mean that there should be a limited awareness of processes and overall architecture.

Reasonable people are often scared off from the technology industry. There are a lot of reasons for this and that could be an entirely different talk.

These people are not a signifigant part of the problem as they can be reasoned with effectively. They’re out there and I hope you can find them.

There are often many solid contributers in successful organizations.

Many of the people I’m about to mention can be effective. I’m going to center on what they’re like when they’re a problem and my take on how to interact with them.

Put things in terms relevant to their interests.

Look out for their ego measuring contests. Outshining them is a sure way to land in their disfavor.

These guys are pretty much irrelevant but common. They are best avoided.

They’re going through a checklist. Give them things to check off and move on to the hiring manager(s).

They want you to sound confident. Very likely to have no idea what you do, why it is important, or how it gets done. They’re looking for you to speak well and sound like you know what you’re talking about.

Sadly too common with downsized efforts, eroded budgets, and no resources to which they can delegate.

They’re looking for someone who can work in a vacuum. Sufficiency is what you need here.

If you’re fortunate enough to interview with a ninja, make the most of it.

BS the ninja at your peril.

Come clean. Tell them what you know and what you do not.

Tell stories from the trenches.

So if we can’t count on insiders to get everything done because the old clue has been promoted or driven out, who’s left?

The previous slides are all mostly hard workers in their own way. The following are not.

Every consultant has worked with this joker.

They can create big problems and large messes of rushed or under-delivered projects that actually have to work.

After signoff, under-baked solutions can be a real operational nightmare. The more complicated and ambitious, the worse the aftermath can be if anything other than ideal.

Can be even less interested in the outcome than all promises sales guy. Relentless in their “buy our stuff. we’re the best” mantra.

When HR doesn’t want to do their job and hiring managers are too busy or not allowed to be involved, the staffing firms soak up a lot of profit by just posting requirements and funneling in bozos.

Getting into a relationship with HR means they can sit at home and capture a significant portion of contractor work effort compensation while adding little (if any) value.

If that wasn’t enough, they also have a profit motive to place as many people as possible, not in placing quality people. Individual headhunters can achieve in extra ordinary ways, but staffing firms almost never deliver in this way.

These people are key actors in the not-my-job industry of lazy.

More times than not, their goals are misaligned to the organization that is employing them. Very rarely is a consultancy interested in solving problems. There’s no profit motive. They’re interested in an increase in revenue and scope of engagements.

The larger the size of the consultancy, the more likely these practices are to arise.

The real magic of the magic quadrant is the ability to get people to pay for the analysis.

Scoped to the average environment in the average business in the average industry.

There is no industry average environment. The best fit for an environment may not be on the leading edge of their wave or quadrant at all.

Yes. You really do have to do your homework.

The classical argument of heterogeneous and homogeneous networks and solutions is usually academic, however interoperability and performance is often misstated or exaggerated.

This individual is the complement to the Industry Analyst. Neither is a replacement for skilled investigation and logical comparison of options.

What is more expensive? A failed implementation following a hasty decision, or a reasoned approach?

My take on how to dig our way out.. but first!

Compliance is a minimum standard, not a gold standard. It is a checklist.

It is not a risk management program or effective governance.

An auditor background and skill set and that of an information security practitioner very rarely intermix.

Harder, Better, Faster, Stronger. This is the way. Always be improving.

Dead Mike knows what was up.

Source. CB4 Video.

Edward Tufte also knew what was up. I’m told that his works are amazing.

Metrics are most effective when cheap to collect and immediately meaningful to the reviewer.

This is a difficult but highly rewarding standard to achieve.

These were some metric suggestions in order to inspire discussion and interaction during my talk.

Some people were pretty heated.

Some didn’t believe that scoring candidates was feasible. It was my contention that academic boards had found effective ways to do just that with their incoming student applicants and surely simple quantitative metric data can be gathered.

One attendee mentioned counting spelling mistakes in a resume.

Another suggested that any metric collection can be gamed nearly immediately. I suggested not disclosing the metric criteria.

If you have your ducks in a row, it will call attention to those that do not. If this does not happen, call attention to it.

Tell the world! Share your data!

When you can rely on data, you can make effective decisions in the light of day based on something more than arbitrary judgement and gut feelings.

When this is pervasive, FUD will be a thing of the past.

Consultants at large failing business are delaying the inevitable unless culture change takes place. The axe man will appear one way or another.

Metrics are factual. They are not slander.

“Oh! Someone might sue you!” That’s what corporate retained counsel is there for. Sharing data in a pay it forward fashion will make the business community and our industry a much better place nearly overnight.

This is important. This needs to happen.

If you enjoyed this talk, you may wish to look at one of my previous talks about security and compliance metrics (a long talk) or the added risks of compliance (a short talk).

Hopefully my sense of humor comes through in this publication method. I attempt to present on issues that I have not heard aired previously in a light-hearted and whimsical way, and only when I feel I can contribute something to the conversation.

Thanks for reading. I’d love to hear from you.

Related posts:

1. ITCi 2007
2. The Politics of Respect
3. What we do

Ivan: County court advocating two year retention

Two year minimum retention. Five years might be useful.

Large ISPs and IP address data:

Identifying an IP does not identify a user

Major ISP: