Bad Penny

I wanted to continue a bit where I left off with a non-technical explanation of what people such as myself do and my commentary on evolving technology management.

Here is the abstract from Unskilled and Unaware of It: How Difficulties in Recognizing One’s Own Incompetence Lead to Inflated Self-Assessments (Justin Kruger and David Dunning, Department of Psychology, Cornell University), a fairly well known publication that appeared in the Journal of Personality and Social Psychology (official link unavailable):

People tend to hold overly favorable views of their abilities in many social and intellectual domains. The authors suggest that this overestimation occurs, in part, because people who are unskilled in these domains suffer a dual burden: Not only do these people reach erroneous conclusions and make unfortunate choices, but their incompetence robs them of the metacognitive ability to realize it. Across 4 studies, the authors found that participants scoring in the bottom quartile on tests of humor, grammar, and logic grossly overestimated their test performance and ability. Although their test scores put them in the 12th percentile, they estimated themselves to be in the 62nd. Several analyses linked this miscalibration to deficits in metacognitive skill, or the capacity to distinguish accuracy from error. Paradoxically, improving the skills of participants, and thus increasing their metacognitive competence, helped them recognize the limitations of their abilities.

This principal, known now as the Dunning–Kruger effect, was given the dubious honor of winning an Ig Nobel prize in 2000. Astonishingly enough, media from the NPR Science Friday show that covered the Ig Nobels that year is still available 10 years later.

It seems to be part of the human condition to have trouble recognizing both competence and incompetence in matters where the observer is not at the top of their game. For example, nearly all Americans and most Swedes think that they are better than average drivers.

This is entertaining and all, but I say this not to call attention to making fun of stupid or incompetent people in the world. Rather, how does one identify and work with people who are incompetent and not cognizant of it? What about if they are highly intelligent and are undervaluing their abilities? How does one know if ones self assessment is accurate? How can one right-size ability and decision making stature? How can one make a qualitative judgement of qualitative judgements?

It’s not a simple problem.

Certainly being a well rounded and traveled individual may help in finding this kind of clarity. This may also be something largely gained from journeyman tradecraft; seeing other methods and masters firsthand. This may be why people talk about risk management as being on par with a JD or MD; it takes a lot of time, passion, and diligence to become and stay competent and aware and literate of the many challenges present in diverse environments and constantly moving technology. The responsibility in the design, management and assessment of complicated systems is also large. Persistent errors here can literally cost lives, crash fortunes, and wreck business models. It may be one of the hardest careers.

The massive efficiency increases to work and leisure of the last 20 years, the strides that have been taken in the knowledge of how individuals most effectively learn, and with all of the information available on the internet, affords the opportunity to learn effectively and in a disciplined approach to become and stay competent in this field. I find that many people make casual reference to Gladwell’s “~10,000 hours = success in a field” rule which seems to be the take-away factoid from his book Outliers. I’m not really sure what to think of Gladwell and his writing, but it strikes me arbitrary. I think of competency in a large and deep field of knowledge to be like snowballs rolling down a hill; they start small, but they increase in size, depth, and have a sense of momentum.

Everyone can’t be a world famous polymath like Leonardo da Vinci or Johann Wolfgang von Goethe. Genius of that level seems to be cultivated only occasionally and is not a realistic role model for nearly anyone. I do know a lot of people that are like Richard Feynman however; a pretty competent man and likely fun at parties.

“A human being should be able to change a diaper, plan an invasion, butcher a hog, conn a ship, design a building, write a sonnet, balance accounts, build a wall, set a bone, comfort the dying, take orders, give orders, cooperate, act alone, solve equations, analyze a new problem, pitch manure, program a computer, cook a tasty meal, fight efficiently, die gallantly. Specialization is for insects.” — Robert Heinlein

I do believe that many of this ages competent [wo]men are involved in information technology and the cream of IT are those that manage complicated systems. Certainly many of the geek cultural heros are, such as the Batman, Gregory House, Ryo Saeba, and James Bolivar DiGriz, show the common geeks aspiration to be widely competent. This may be the true root of what geek culture is all about.

Predictably to those familiar with my past work, this leads back to my usual harping on metrics and data analysis instead of so called best practices, third party industry rankings, or arbitrary standards. Meaningful data is the only way to get away from the constructs of cognitive bias that lead people to fear plane crashes and terrorism but feel completely safe driving a car in nearly any condition.

I pose a question: can the competence of a complicated system be left to a specialist who is not competent to judge the quality of all of its components? The stakes are high after all, for what could be more important in a world that runs on the power of information but the secrets of powerful people and organizations? I contend that a conformist drive to make industry into a uniform, standard, and specialist product is directly at odds with that of producing quality work product for our industry as a whole. The business would prefer that an employee can be easily recruited, fired, or trained to perform this job, but I do not believe that this is a realistic goal for risk management specifically, and perhaps, IT in general.

In reading one of Andre’s dives into the waters of competence and speciality, I was stuck at his harsh but close to home hitting words about the state of the industry and professional credentials. He invokes Dan Geer. Here is the preface to the talk he cites, a keynote at Source Boston:

Good morning. If you were to come to my office, you would see on the wall these four rules:

• Work like Hell
• Share all you know
• Abide by your handshake
• Have fun

And later:

Only people in this room will understand what I am now going to say. It is this: Security is perhaps the most difficult intellectual profession on the planet. The core knowledge base has reached the point where new recruits can no longer hope to be competent generalists, serial specialization is the only broad option available to them.

Ah there it is again. Serial specialization might be closer to what I mean, but it is a bit hairsplitting to argue about if a skill set is generalist or serial specialist. I think the real criteria here is passion for learning and a will to do what is necessary to learn and work well.

I was discussing this with a friend who is a developer who has been kept pure from this circus but hears some of my tales. He had this to say when I asked him about his experience with efforts to make a homogenized software development environment. He asked that he not be named but his words from the developer side of the house sounded quite familiar to me:

Having worked in a variety of industries, at a number of companies, provides one with a reasonable substrate against which to compare and contrast the natures of the environments in which one works. A set of patterns emerge that reveal the nature of an organization. Unfortunately, many of them reveal weaknesses or inefficiencies in the modern place of employment.

College degrees are all but an afterthought nowadays. Employers have responded by creating a set of artificial restrictions. Since I started at my current employer, a policy has been enacted by the HR department, stipulating that an applicant seeking, for example, a software development position, must hold a degree in computer science. This is a terrible idea. What does HR know about finding talented programmers? Perhaps the most talented programmers I’ve worked with have been the ones that do not hold a computer science degree. A couple of them never even finished college. An ability to excel in software development is not restricted to those who have spent years and a bunch of money on formal training in the field. As with most other crafts, a good indicator of success is the zeal with which one pursues it. Why not allow those who are responsible for developing the software determine the qualifications for those who will ultimately be hired to help build it? Senseless policies like these disempower those accountable for the work and serve as a blockade to workplace efficiency. It’s not much of a stretch to feel like HR, in situations like these, is a congressional body for the workplace, passing more and more laws as though it’s an effort to convince others that their existence is justified.

Something else that becomes clear after working in various employment climates is how important it is to hire really good people. This sounds like an obvious statement, but chances are you don’t even know how much this affects you. Hiring great people is what allows a company to trust employees to be able to do their jobs with minimal intervention. Being able to trust employees in this way reduces the need for artificial barriers that stand in the way of getting real work done. These policies are often in place to protect a company or workgroup from someone doing something stupid. Employers should aspire to hire good people–the best they can–across the board, so that they don’t have to annoy the good people with policies or artificial divisions of work responsibilities that mainly get in their way. This is, of course, easier said than done, but if this approach were more prominently represented in hiring practices, I suspect we’d all be better off. But, hmm, where would everyone else work?

Practically speaking, it seems that the companies that are known for difficult interviews are the ones that have a higher caliber of employee across the board. I’m not talking about making interviews hard and annoying for its own sake, but rather interviewing “smart,” and determining a good fit, not necessarily who has paid the right amount of money for professional training.

Additionally I would like to mention that with the great responsibility and knowledge required to perform well in these critical roles, there is the sticky point of ethics and professional due care. The current industry standard of “not getting caught being a jerk” is pretty weak. This relates to my point of competence and generalist/specialist bias as there is a classical example that could be considered as a useful metaphor: Bushido. Will has his own rant on the subject which I recommend.

In the end, the world is wide and filled with wonder and things that we as humans will never fully understand. I would suggest, as an ideal, not to fear the unknown but appreciate it for the challenge that it is to do something that you haven’t done before. There will always be people who have something to teach you. Learn from them.

At the end of the day, it is the seasoned veterans from the trench warfare of operations and gladiatorial arenas of the boardrooms that will steer a ship to calm safe waters. They have the best stories, have seen the mountaintops, have looked into the yawning abyss, and have come to you to tell the tale. Are you ready to hear them?

Image References:
Creating Passionate Users
Dave Gray