As most techsec professionals, I’ve been asked to do more and more in application security matters, an area that I’ve usually seen specialist ninjas dominate due to the often extreme technical depth required of the realm.

I’ve written previously about how [in]competence seems to be very hard to self-identify in most humans even when not dealing with areas of high complexity, so why not have some quality guidance so that one does not sound like so many of our kind; talking out of their depth and giving bad advice.

So okay.  How far should someone who wants to remain generalized, but wants to be able to offer some kind of limited competent analysis and dialogue on application security, go without going off the rails into overreaching bad advice?  Appsec is pretty much the biggest deal in the information security field since everyone adopted garbage technologies as lame development platforms a few years back.  Writing secure code has always been important, but things used to be a little different.  The appstack is now wide and deep  and, if you own the appstream, you can pretty much access/subvert/rob/blind everything that matters in most environments.

What I should really do, I thought, is ask for advice on how much confidence one who doesn’t specialize in this area should reasonably expect to have when putting together solutions to solve problems in this realm where the best experts spend most of their waking lives knee deep in it and I’m all over the place.  I thought that who better to comment on the reach of a generalist in this area than a ninja who specializes in appsec and fights with the other experts on public mailing lists.  To this end I asked a friend of mine who is one of those specialized appsec ninjas what he thought a generalist should know about application security, how far they should go when they talk about about subjects therein, and what were the most important areas that they should be familiar without making themselves part of the problem.

Not surprisingly, he had a lot to say on the subject.

With my  background as a sysadmin, network engineer, integrator, enterprise implementer, general technology consultant, and process/workflow/control/whateverist, people have come to me to solve complicated problems on a routine basis.  The most interesting of these scenarios tend to be when I’m presented with a half-solution from very competent people based on how they would approach a problem, but leave the details to me to pull together and fit those square pegs in their matching round holes.

Big developers tend to think that their skillset is the pinnacle of all technology competence and nerd achievement and also, perhaps secretly, all other technology people are aspiring to be them.  If you have visited at any big dev shop, but especially Microsoft, Amazon, or Google (and I have), you will have seen this first hand.

Additionally, might also discover the same god-of-all-the-lesser-nerds mindset with many talented high end network architects and nanog-type network engineers, SAGE-type big bearded UNIX admins, and other specialized people.  They, perhaps, under-appreciate that there is a lot more art and craft to managing complicated systems than just writing code, routing packets, or server uptime statistics exclusively.

To review:

• As a generalist, how far can I go in being effective in this space?
• What are some good authored sources of methodologies that people can read for them to begin thinking about how best to approach this iceberg situationaly.
• How is it best to employ experts effectively: how to pivot from analysis of unit test results, running automated processes using assorted codebase analysis tools, and how far should a generalist go in  performing the analysis themselves before punting to a specialist?
• What would you have to say on the balancing act between developer time and secure coding practices; making the business case for quality and developer maturity instead of an absolute ship schedule.

This principal from a major application security SaaS vendor had this to say:

Network sysadmin types with those backgrounds and that have limited experience with any programming language including Unix shell, BASIC, LOGO, or hardware robotics/electronics and that can’t understand concepts like recursion or loops simply not need waste their time touching anything related to appsec or pen-testing.  This is inclusive of pen-testing, social engineering, as well as appsec, and even physical pen-testing. Social engineering newbies have Metasploit and/or SET as the #1 tool in their toolbox, even if it means paying $15k to Rapid7 for the Pro version of Nexpose and some mid-to-senior level pen-tester a few grand to automate SET with a custom GUI.

THEY STILL NEED TO KNOW BASICS!

That being said, I know very few network or sysadmin or even audit people that want to do pen-testing or appsec that DO NOT have at least a little scripting experience. A little scripting experience is going to get someone reasonably far in pen-testing and appsec.

A word on the tools: they suck.

Even when you run all of the commercial ones, they still suck in the wrong hands. They say a fool with a tool is still a fool and it is 100 BILLION PERCENT TRUE. I do not want to cite specific people in our industry, but they exist. They have social engineered companies who want appsec and they run every commercial appsec tool (all app scanners, fuzzers, network vulnerability scanners, and static analysis tools) on everything they can get a URL, build button to drop companyname.com on, or box to “make all” with. They are bunk, but people keep buying their garbage, which in turn gives them more ability to buy more tools, which makes everyone in the appsec industry look bad.

Then you’ve got other bunkos that have a SaaS offerings, who hasn’t updated their product in 5 years. It’s a perl script that fault-injects HTTP parameters and pumps the HTTP responses through a massive regex. Then, they have 19-23 year old kids with no degrees and no infosec experience, but perhaps some QA experience at A Major Game Company for three months, and verify the true positives by running through checklists. The concept has been tried in various other organizations and it is well known as a “scan factory” as if that’s somehow a good thing.

Clearly a dig at people who know who they are but no one else does.  I love it when people do that.

Again, the tools blow. You have to use your brain even if you are writing all of the tools yourself or not. It’s the way that you use the tools or your own tools is what matters. Training kinda helps, but the problem is that SANS and Ethical Hacker programs just tells people the names of the tools and the slightest introduction to them.  Really, they tell you nothing about how to use them. Nobody tells anyone how to use the tools.  It’s simply not written down or available on YouTube. Offensive Security has some cool training for network pen-testing, but they don’t really get into appsec proper.

The best training on appsec comes from Aspect Security and by reading the beejesus out of the OWASP website (and going retarded on the OWASP community, both online and IRL), and perhaps memorizing the MITRE CWE/CAPEC. Perhaps combined with a good pen-test mentality (from Offensive Security and learning how to use the tools on your own by playing with them on test grounds or real-world environments i.e. YEARS OF EXPERIENCE), this can make a ship leave the dockyards.

If you want to set sail with appsec, you are in for quite a disappointment.

NOBODY gets it. NOBODY.

You can memorize TAOSSA and WAHH but this isn’t going to do anything for you. You can write these books and still be totally clueless when it comes to fixing root causes and root issues in our industry. This is where even the best and brightest fail.

Anyone that’s done their tenure at Aspect Security is very rooted in awesomesauce, and certainly there are other people that kick all of Aspect Security’s butts, but they are one-offs and usually esoteric specialists in some arcane appsec art form.

I’ve heard examples made of people who everyone knows who specialize in XSS, SQLi, know everything there is to know about code audit in a particular favorite language, etc.

There are no appsec generalists. There are some super rockstars, but they are too busy at kicking ass and taking names to talk to anyone. So you have to totally learn this on your own.

YOU ARE ON YOUR OWN.

I suggest just becoming a rockstar. A search for “application security” on books24x7.com lists over 1500 books. I suggest reading all of those, and then doing the same on Safari Books Online. It should only take about two years of straight reading and doing nothing else to accomplish this, so it’s not that hard because it’s just work on your eyes and you can get a Kindle if your eyes strain too much. You can glue it to your stairmaster controls and stay in shape at the same time while reading all of them!

After reading and knowing where to find the info you’ll need when you get stuck performing real world bid’ness, download every testing ground and every tool and use everything on everything until you get results that you like. This is likely going to take at least five years, and you’ll also probably want to be testing real world things during this time, too. Think of it like earning five different CCIE programs while being the only guy running the BGP traffic engineering at Level 3. You have to work sick hard and play around with things a lot until you get them right.  Also, you’ll want to use things like statistics and math (at least a little bit) in order to judge if you are going in a good direction and are being successful.

Using metrics and statistics to assess success?  Hooray!

Finally, you’ll need to write your own tools, tool wrappers, and books. It’s like studying for law school finals and having to outline every chapter of sixteen four thousand page reporters, which basically means you’re writing a four thousand page book that summarize the sixteen reporters and the lectures and work you’ve done, but really it’s all about getting into the psychology of the court cases and who was involved and trying to pretend you were in that court room.  Legal metaphors aside, capturing the mindset is key.

But there are some shortcuts and things I’ve learned to help others succeed. It actually helps quite a lot to know something about their background and what they are capable of doing. Becoming an appsec (or IMO, any infosec) expert means being a trusted adviser to an organization.  Fulfilling this need requires multiple people in multiple roles on multiple teams.

Yes.  Teams.

Building your dream team is basically what this is all about; you can’t be a lone hacker cowboy. It’s not like that. You get a tiger team like in the movie Sneakers. Robert Redford did not do it without that crazy blind guy and Dan Aykroyd as the conspiracy nutball.

More over, in reality, the leaders are never as good as Hannibal or Redford. Everyone has to pull equal weight. When you work for a CISO, you need to have the same title as everyone: “Assistant to the CISO”. Flat hierarchy. Everybody can probably do some of everything, but people count on each other to get their bit done and done right the first time and all that jazz.

I usually see a random title and suffix of working for “the office of the CISO” when I run into members of these teams.

So when I say to a team guy/gal, “I am going to help you with appsec and pen-test,” it means a ton of loading on them is going to happen. I am going to assume it’s someone like me 7 years ago, or maybe someone like Ian Gorrie, or somewhere in-between but also representative of others skillsets that are somewhat like us — and probably as cool, but probably never going to be a ninja or super rockstar, even though Ian and I might be that cool someday.

I think I’m alright at some things.

On fundamentals:

Basically, if you really want to get good at appsec, get good at security first. Everyone else keeps saying the other way around, but I disagree. There are certain things that need to be learned in the right order. Starting out with tons of network security experience is good, as long as network security is not the end-goal of this person’s learning pursuits.

That said, a person needs some serious investment in the instructional capital necessary to get up to speed. The Secure Programming with Static Analysis book is not enough by a long shot. The Web Application Hacker’s Handbook 2nd Edition is merely scratching the surface, however the accompanying labs at mdsec.net are basically a must-have experience for getting started with app pen-testing.

Here’s the primary 5 investments necessary to be a good appsec dood in 2012:

1. Appdev.com Java landStart with Java EE Programming: Servlets and JSP Fundamentals. Build all of the projects and code in the course. Supplement learning servlets, JSP, JavaBeans, and JSTL with other online or book resources.  Move on to the servlets, JSP, and JSF content. This content will go further past the MVC model and more into framework specifics like JSF and Struts.  You’ll probably want to take all of the additional courses in the Java Enterprise section, but be sure to take the XML Development in Java before hitting the EJB or Spring stuff too hard. Supplement it with the Java Brains free online courses. Do not pass up the Hibernate courses before moving on from the Appdev.com courses on Java Enterpise, make sure you have a solid foundation of knowledge of the JAX-WS stuff. You are now nearly ready to assess the OWASP Webgoat and GoatDroid codebases
2. Play with the OWASP Webgoat and GoatDroid implementations to understand them fully. Note that you’ll want to import these into many build tools and environments — not just Eclipse and Jenkins, but also IntelliJ IDEA, Netbeans, and many other CI servers.  Find other code to build. Build it in every build tool. Being a build and release engineer is a primary skill set of any appsec professional worth their weight. You’re going to get 100 questions on the first day to any client site about this kind of stuff. You have to know your stuff when it comes to building and releasing Java Enterprise code.
3. Rock on with the PluralSight learning offerings. I suggest to start with the Android courses in order, and then move on to the iOS learnings before anything else.
4. Check out raywenderlich.com and other sites like it.
5. Build your own projects and then tear them apart.  Mobile apps and the web services that they consume are really paramount in 2012. They are our new low-hanging fruit and they are important. Expect to see more and more use of self-modifying/checking code everywhere (mobile/cloud/web-service apps). You’ll need to know how to deal with this stuff at some point, so you might as well start learning now. Get yourself a copy of Surreptitious Software and Practical Malware Analysis after dusting off your IDA Pro Book kid-gloves.

A special note here: Javascript and HTML5 are insanely good investments. haXe will continue to be a niche language that is “nice-to-have” because of its future-looking capabilities. John Resig recently stated that Khan Academy will have most of its programming courses in JavaScript. This does not necessary mean that Scratch (really Squeak which is really SmallTalk) isn’t a useful or better first programming language. People need to learn the basics of OO, especially once they reach a certain point. Someone who has followed my 5 step approach above will be at this point somewhere near the middle or end (depending on how much previous experience they have).

I highly recommend Object-Oriented Software Construction as well as Core Security Patterns at this time in the career. Almost everything built should be in Java (or JSR complete language such as JRuby or Jython). Yes, you will also want to know enough Obj-C to strangle yourself, as well as glue frameworks such as PhoneGap. Don’t bother with MonoTouch, at least not yet. We’ll get to that.

There will be a new book on Web Security Patterns around June 2012. This book or one like it will probably be pivotal in the way that System Assurance: Beyond Detecting Vulnerabilities should have been. Eventually, some developers will start to “get it” when it comes to appsec problems. They’ll put in the work ahead of time. Most won’t. Most will try to hide behind seemingly secure frameworks such as .NET.

.NET will remain a revolutionary secure approach to handling the appsec issue. This is why you must learn it last. Not only is it very secure out of the box — almost all of the issues that require looking at during a pen-test and/or secure code review involve intricacies that can only be learned after Java Enterprise, Android SDK, and iOS SDK are learned. I also highly suggest learning .NET with Mono instead of without. There are you many benefits to doing this how I described, but to put it in language that you won’t understand — let me give this retarded analogy: you learn to play the clarinet before the flute. You just do because that’s how it’s done. You wouldn’t want to learn the flute first, because that would mess it up.

So, there you have it. Learn Java before .NET. It will also be easier to break things when you use a system that is easily breakable. Using this logic, it may be best to learn Perl before Java, but really Java is the most important because anyone who does any static analysis will be focused primarily on Java Enterprise.

Most other things to learn are more generally a waste of time and should be considered optional components in the learning ladder. For example, there’s not a lot of reason to learn Struts2 if you learned Struts; or to learn iBATIS if you learned Hibernate. You can deal with those problems when you run across them in the real world (that is if you run across them at all). This also brings me to discuss things that developers don’t know that are there, but these things are really there and you do have to learn them. Things like memcached. Good developers do learn these things, and as a security professional, it makes sense to stay at least one step ahead of those geeks. So learning the external components that connect apps is paramount. You’ll need to read a bunch of Oracle, TIBCO, TeraData and similar books, as well as get some hands-on with these components — at least in a lab or wherever your build environment is for learning purposes. Try to get as many of these unique solutions into your head such as cache, cluster, and grid technologies.

He had more to say later about appsec in general and if one should make the extreme effort to be a practitioner, he provides more detail about what specific things should be done and learned.

Dood. It’s easy.

#1 : Don’t be a douchebag.

If you really can’t code at ALL, especially on ANY/EVERY platform in nearly EVERY language — don’t bother doing appsec. I don’t mean you have to know the differences between assemblers, but it would be nice to know who or where to go to if you need immediate answers to programming questions. Try like SafaiBooksOnline, Books24x7, Ebrary, a digital library, etc. Build a meta search engine and your own dorking libraries.

DON’T sell yourself on being an appsec guy like a douche who doesn’t know that he can accomplish the same from a Python script as with a Unix or NT shell. (See also most QSAs)

Complaining about he general badness of auditors has been done before, but since so many that I run into view assessment = audit = pen test = appsec (which should pretty much deprecate any other opinion they might deliver after such a claim) it bears repeating.

#2 : Learn HTML, XML, Flash, Flex, AIR, Web Services, SQL clauses, SQLi, Javascript, XSS, CSRF, Ajax, RIA frameworks, Node.js, MongoDB, CouchDB, Project Voldemort, et cetera

Best to know these before you know every detail of HTTP and SSL/TLS. Too much network turns you into a network guy. At some point, try to learn about HTTP Header Injection or SSL vulnerabilities (especially if you get stuck or bored on any of these other suggestions), especially since header injections involve HTTP response splitting and file download injection vulnerabilities.

I suggest you bust out the xmllint shell on HTML and XML files, including live sites via HTTP. Kill two birds with one stone. Libxml2 comes with xmllint(1) and is installed under Mac OS X and probably your Unix of choice (e.g. Linux, Cygwin, etc) or can be within seconds or minutes unless you are on a soft keyboard or your keyboard and operating systems are in a different language. If there are real life distractions, then you need to get rid of them.

Download SWFs into SWFScan. It’s free. You like free. You might also want to automate with Nemo 440 or check out the commercial Flash disassemblers/debuggers. You’ll be right on your way to FlexBug in no time! There are OWASP videos on Vimeo. They talk about stuff that you should listen to.

Learn how HTTP communicates inside an Enterprise. It’s usually cleartext REST or just plain XML serialized data (DEFLATE compressed) that is sessionless — it’s so easy that you don’t even need Firesheep. At this point you might want to learn everything about Base64, compression, and light forms of encryption and session management (besides HTTP session management).

SQL has clauses. Know which ones you can inject into and why. Know it for every popular language and RDBMS. Read a few hundred SQLi cheatsheets. SVN down the latest sqlmap, run it everyday, and read the code. Know how to save an HTTP request to a file and automate it. At the very least, run Havij Free Edition through Burp or Fiddler and reverse engineer what it is doing so that you can learn how to do that stuff. I want to reverse engineer the commercial copy of Havij because it claims to support SQLi through ModSecurity!

Learn Hackvertor and study Javascript obfuscation/compression/minimization. Good. Now learn those concepts without Hackvertor and Javascript by implementing your own Javascript language that’s exactly like Javascript and Actionscript. Study the OSSP C-based interpreter source code and think about what it would take to build a Javascript fuzzer. Keep learning more Hackvertor this whole time. XSS will be easy to understand by this point. I suggest you also check out Casaba x5s and all of the work by thornmaker, .mario, and sirdarkcat. There’s also cool tools such as DOMScan worth checking out.

The rest will be easy by this point. If you can write a Javascript fuzzer, you should be able to also write one that targets unmanaged code. Now you will want to learn how to put backdoors in managed code, including their frameworks. Learn the callbacks and interactions between all of the major VMs with various kinds of code e.g. Java Enterprise, Dalvik, C# ASP.NET, PHP, etc. Learn how to “fuzz” VMs with various code such as running/compiling anything with Parrot VM, as well as the weird stuff you can do with JaCIL, a CLI to JVM Compiler (or IKVM.NET). Learn Reframeworker.

At some point, should you be looking for other ideas: learn PHP security. PHP is very popular with people who don’t have time to learn things right. It’s also fortunate (or not fortunate depending on your perspective) that PHP is highly configured and lots of application security mistakes can be made in the configuration of PHP in addition to the code. This is great for those with a LAMP background.

# 3 : Learn App Pen-testing

They love it

If you want to do pen-testing, please try not to get suckered into the whole ego trip of it. It is cool. Your mom even thinks it’s cool. Hot members of your sexual preference think it’s cool. My girlfriend thinks it is cool but ignores me. There’s nothing wrong with that. I kind of like it that way.

Pen-testing is a joke. It’s cake. You steal all of the ideas that laresconsulting.com has presented. You combine them with your SQLi, XSS, and read/write inclusion knowledge. I would probably use Core Impact if I oversold non-appsec-pen-tests to clients.

Appsec is the bottom of the vomit bucket. Every target has an XSS or a SQLi — or SOMETHING ELSE. Load up XSSF or sqlmap-dev and get to work. But first — you need to do what laresconsulting.com cannot do: find an XSS or SQLi in the first place (let alone that SOMETHING ELSE, which is usually a read/write include attack, file upload attack, business logic flaw, etc). For this, you only need to do a few things. I’ll run down them here as fast as possible, but it will be confusing. I don’t want to make it too easy for you. Run Netsparker CE in Fast (No JS, max speed) mode through Fiddler. Configure Watcher to process sessions offline when you are done. Look for High Issues labelled User Controllable HTML Attribute. Double click them. Configure x5s (you have to learn these tools a bit first AND you have to know XSS and SQLi). Replay the requests in Fiddler a few at a time until you hit XSS jackpot in x5s or XSSRays. Start emailing targets with URL shortened and HTTP/JS obfuscated links that go to XSSF. Exciting. Next you need to find SQLi, which Netsparker CE can do against MySQL or MS SQL (sometimes). Burp Pro Scanner will find SQLi in lots more things than Netsparker CE (focus on the user controllable suggestions from Watcher first because you’re sooner to hit jackpot than with zero knowledge), but the cool thing about Burp is the ability to play around, view things (it’s a good idea to know EVERYTHING about the data going into Burp so that you can carefully play out decisions on what you are going to send in HTTP requests — Burp Pro has a Search feature btw), tweak the settings, and automate.

Browser tools:

I suggest the Multi Links and All-in-One Gestures add-ons for Firefox, especially good when combined with the Web Developer Populate Form Fields drop-down or FireBug. I also like Fireforce with the skullsecurity lists (rockyou-75.txt is my current favorite, but I tend to mix it up with L517 and interesting related data). Fireforce is useful for when you don’t have a login, and haven’t yet stolen credentials with XSS or wherever else.

You tried to login with SQLi, right? If you use any other Firefox add-ons, then you’re an idiot. Learn how to do it with a bookmarklet. Also see rule #1: Don’t write a 40 line Python script to do what you can do on-the-fly with a 14 character Unix or NT shell-fu command line. Burp will be able to handle all of that. Repeater is your new browser.  You use Proxy History and the Comparer tool, too. Or Intruder. I use all of these tools. I hate Target and Spider and you will, too. Better to automate stuff faster using APIs and fast things you find around. I am guessing that O2 does this probably ok, but I’d probably just figure out how to do it at a Unix or NT shell (wherever I happen to be at the time, although as a side note, it’s nice to have some statically compiled tools for various platforms especially something like that previously mentioned Parrot VM). I really prefer to have Ruby, PHP, Python, bash, Perl, OSSP js, OpenSSL, and a few other things around whenever I’m on a command line, especially calculators like bc(1) and dc(1).

It’s nice to know more than a few interpreters by heart. You might want to learn some statistics and encryption along with your shell-fu. If you can’t figure out a potentially vulnerable URI with potentially vulnerable insertion points (parameter names, parameter values, URI/REST/arbitrary parameters, known or arbitrary cookies, known or arbitrary headers, etc) using Burp Pro, then you may want to go back to x5s. I prefer attacking insertion points that I know affect the app itself, which may require a lot of workflow using your browser. Work for it! Use QA techniques such as equivalence classification and pairwise testing to provide some focus and attention to what you are doing. Get some books on exploratory testing and apply the techniques to security testing. Make sure you understand your target environment the best you can with the resources you have (also see laresconsulting.com techniques or similar). Yes, RDP hijacking is cool — especially when you’ve got Forest Admin and recreate what the current NT admins think is their Domain infrastructure. What would be better would just be to have system access to all of the target’s public-facing RDBMS servers — and an XSS Proxy or XSS Tunnel inside a few key employees’ browsers, so that you can constantly spit DriveSploitMSFbrowser_autopwn Fast-Track fu all over their target OS environment. You probably got RDBMS server access through a UDF injection or silly stored procedure, didn’t you? UDF is a cooler acronym than RDP. XSS is a lot cooler of an acronym than most people are willing to credit it.

Remember, laresconsulting.com has to do a lot of pen-tests local (e.g. in the target’s Porsche while sitting in their parking space). You can do this stuff from space.

–update–: Ok, as for the above, I barely do any of it anymore. I just don’t have the time while on assessments. I look for higher value AuthN/AuthZ targets most of the time. For the basic bug checks, I just slam the host with RAFT and SVNDigger lists (directory config and lists first, followed by files) using a properly configured DirBuster. While that’s running, I prop up SearchDiggity on the site if it’s an Internet one (or do similar digging myself if it’s something I have to Intranet or VPN into). Best of case scenario is that I have root access and can grab the source code, throw it into Fortify SCA (or run SecurityScope), and/or run audit tools, especially cvechecker.

I also tend to look for high value vulnerabilities first, and try to fit parameters to them. For example, if I see a “/” in any parameter values, then I hit them with path traversals. If I see any “orderby/desc/asc” parameter names, then I hit their values with SQLi. Anything that includes user-controllable HTML gets some custom XSS checks by hand.

Final point (since I get source code a lot) is that Fortify SCA can be made to easily do things when you specify custom analyzer views with the taint:web and diagrams showing.
–end update

A PHP example:

When you hack your PHP-dev-would-be-friend, jump on their box and install modsec, ossec, and set them up together — probably get them talking to OSSIM. Whitelist those parameter key names and their reasonable values, as well as any other insertion points. Whitelist URIs and REST parameters if applicable. Setup ossec to monitor the integrity of the frameworks, containers, content, et al. Show your PHP buddies how to run fimap. If they are total webnerds and can’t do command line, get them going with Arachni’s web portal and RIPS web friendly thing. If you have money, probably get them HoneyApps, Metasploit Pro, Netsparker Pro, and some WhiteHat Security Sentinel API action (for things like HTTP Response Splitting testing). Go back to your days of installing rpcapd on everything. Import Netsparker Pro and pcap data into Metasploit Pro.

As either attacker or defender, setup some nice Fierce-v2, nmap, Medusa/etc, Nikto, DirBuster, skipfish, and OpenVAS automation with any of your other existing tools. Sometimes these tools can be wicked fast at spotting something that you want to find. Or stuff like GoogleDiggity/BingDiggity, SHODAN, et al.

Lots of this type of automation.

# 4 : Learn how to go from a bunch of ownable code to a bunch of less ownable code.

The key here is to be a total badass with every static analysis tool. You won’t be able to do this because you probably didn’t spend enough time learning the language callbacks to their frameworks. Without callback knowledge, you’re kind of stuck — so I’d recommend learning Ounce [Appscan] first if you are stuck in this rut. Unfortunately, getting Fortify means you’ll have access to tons of languages you probably won’t use. Ounce is cool because you will get information about lost sinks. Other tools are less common, but people are badasses at all 4 of these: Ounce, Fortify, Armorize, and Checkmarx. You could probably run Klocwork, Coverity, GrammaTech, and others on Java Enterprise code but probably not C# or PHP. You’ll learn to use fast techniques to scale and learn these, but it takes time. Nobody writes about it, but Cigital, Gotham Digital Science, iSecPartners, and Stach & Liu seem to know way more about these tools than others, including Denim Group and AsTech Consulting, or ISS for that matter. Heck, I’d say those 4 know more about the top 4 static analysis tools than the vendors themselves. It’s similar with Web Application Security Scanners — but in this case, the free one works better than most of the best commercial ones. Remember that this technology isn’t rocket science — it’s a satisfiability solver or perhaps an automated-theorem prover at best, but usually just mostly (lots of adverbs here, sorry, but it’s true) an AST parser.

This gets really annoying, so the first thing to do is go to ohloh.net and find all of the largest open-source projects, especially ones with lots of developers and lots of churn, and build them in every available build tool for that project or language(s) that the project components can be built with. Then, try your own free AST parsers on them, such as all of the ones found in Yasca (but try tons of static analysis tools). NCSS is one nice thing to know for scalability purposes here. If you can get yours hands on very powerful continuous integration tools, I think that will help your cause — but the important part is to keep building code, especially code under constant churn. Every directory on your OS should be linked to a GIT or SVN repo. Maven should be automated like mad.

I’ve gone a few different directions on how I view security testing for developers. I have always felt there would be value to adding fault-injection checks to HtmlUnit or a Javascript or Flex/Actionscript component testing framework. However, there are a lot of subtle flaws in web application security that I think are best demonstrated in Burp Pro by saving a session file of just the Repeater in order to share it with others. The SAZ file in Fiddler is another option — it’s cool that Fiddler can also output XML and Microsoft Test Professional test cases. I am inclined to think that these are great tools for pen-testers, quality testers, and development testers — they will have much more lasting value than custom integration scripts.

# 5 : Start combining everything.

Mix it up. You’re an AppSec DJ. Girl Talk will say your mashups are a bunch of crackups.

I decided to update this again. I am installing RIPS for PHP today. I saw it in action earlier today and compared the results to the most recent versions of HP Fortify SCA and Armorize that were tuned by Vinnie Liu and his team.

I am in shock that an open-source web portal security-focused static analysis tool would BEAT OUT the pro commercial tools categorically. It hands-down took the first prize by a million points.

–update-– Fortify has since gotten a little better, but RIPS is still quite good. It failed to run on me a few times lately when Fortify worked.

Another tool that works primarily on runtime, but also scans PHP (and CFM, JSP, JSPX, ASP, ASPX) files is inspathx. It is in development and I am working with the developers of it. It’s open-source, command line, and written in Ruby.

Arachni, another open-source project written in Ruby, will have a web portal that rivals the HP Assessment Management Platform (AMP).

Between RIPS and Arachni, it appears that sysadmins, network engineers, network security types, and SOC people can now help out and/or learn application security including code review.  I would also combine these tools with ModSecurity feeding OSSEC feeding OSSIM (in addition to other hooks in OSSIM such as PADS, Snort, and Nessus/OpenVAS).

—update– this paragraph above is still fairly good, although I have heard of some improvements to log management like components in the past year. Metasploit Community Edition is also really nice

On working in enterprise environments:

So… um… it’s apps. XML is like 51% of data exchanged by apps in enterprises and it’s a lot like HTML. So learn that right away! I suggest if you have a Unix background, install libxml2 binaries (this is default on like Ubuntu, Cygwin, and Mac OS X I presume — or it SHOULD be) and type xmllint. Then read the –help and man page. Then start playing with it and XML and HTML files. You can even feed it a URL and drop to a shell. I recommend doing that first and using “help” inside the shell and you’ll figure it out real quick. Then learn to program more with XML — there’s a few other ways of doing that CLI with like xmlstream or some other lame commands, but you’ll probably want to learn lxml for Python or whatever language you want to learn or already know. I highly suggest the Refactoring HTML book for learning this too.

Then you can go in multiple directions with that XML/HTML knowledge. You can use it for code knowledge, server configurations, as well as reading the crap in HTTP responses. HTTP is pretty big. Besides the Offensive Security network pen-test stuff, you’ll want to enhance your ability to apply this to the apps. Sure, SSL is important but ssllabs.com almost precludes getting too fancy with the openssl CLI or stunnnel. RFC 2616 and related are penultimate to understanding HTTP, but really you want to learn the web proxies, especially Fiddler2 and Burp Suite Free/Pro. You can skip this knowledge and become super elite, but why? Web proxies and MITM proxies in general are awesomesauce for speaking language with others and showing them things. The Microsoft Press Hunting Security Bugs book is great and it comes with web-based companion content. Really, you want to master something like Mallory instead, but a broad knowledge of all types of Web/MITM proxies is going to be a boon. You’ll probably want to learn uhooker, Echo Mirage, and WPE Pro as well while you’re at it. ImmDbg/Olly or what not might be great. You can play with protocols like an appdev designing and implementing the protocols in apps or better. It’s RE awesomesauce. There’s not too many good classes on RE, but I’ve seen the SANS material and it’s not horrible. Their RE and adv exploit dev classes are much better than their webapp pen-test classes. There’s a million ways to skin these cats and I recommend learning a bunch and moving on to getting closer and closer to the apps you intend to understand.

Since you got HTTP under your belt, try out the different test grounds and real-world apps. See the names and types of parameters and how they work together — their associations to page and other parameters. Try to understand how the parameters might work on the backend, whether a web service, other service like LDAP, XML query structure (e.g. XPath, which you can also learn via xmllint), database, or files. Fall back to that SMTP know-how you learned from the Offensive Security stuff — email is common in app forms and is heavily service-oriented. I think it really helps to learn basic QA 101 here — knowing how to not repeat testing the same stuff over and over (removing equivalent classes, or recurring idioms — to say it another way). Know how to get to the “meat” of the problem that you are trying to solve. This is your brain at work! Combinatorial explosions are a neat concept — it’s like mashing buttons and finding awesome combos! Mash up some parameters and see how they go. Start with things like login and password fields. Sometimes the authentication is done via a single query string, therefore you can combine stuff like ‘ OR 1=1/* (for the login) and */– for the password. Understand that if you want to do security, there are no shortcuts and that everyone MUST be tested, but realize that some kinds of testing will give you quicker results (and better/faster understanding). Build some fuzzers and fault-injectors now!

What if someone doesn’t want to spend a decade learning all of that?

There are some really cheap wins in appsec that just brute-force awesomeness. Like ProxyFuzz. IMO, some of the webapp fault-injectors are classes above the rest, and the ones that are more actively developed are nice, such as arachni or w3af. However, stability can be a super differentiator, which is why many people rely on tools such as Burp Pro and techniques as seen in Gray Hat Python. The funny thing is that even these are not stable — you have to gauge their stability at handling any one particular task depending on a lot of factors. Knowing how to scale and do network/system performance stuff can really help here. I wish I used iftop more to see how fast any given brute-force, fuzz, or fault-injection tool is performing (and we all wish we could just test everything from a localhost — not a bad strategy.  Often RE’ing and making your own version of something in a lab is faster than testing it in a prod, dev, or test system). Ease-of-use combined with power are also nice. It shows in all versions of Metasploit and definitely in Burp Pro and Netsparker Pro. I fight the typical mindshare/mindset stuff usually, and perhaps I’m wrong about that, but Appscan, WebInspect, and Codenomicon just don’t do it for me when the free Wapiti and skipfish + the cheaper Netsparker and Acunetix seem to do so much better with the testing grounds and real world apps.

Static analysis is straightforward: one loads Fortify into Eclipse for small-to-medium sized projects that involve almost all code except corner cases or unsupported languages. Languages like SFDC Apex are best done with Checkmarx, especially when there are just snippets of code and not a buildable system (good for cloud based apps). Knowing how to build the code is important. Know your Eclipse/Ant and VisualStudio/MSBuild well by building tons of code — easy to find in the world of open-source. Especially try large or difficult projects which you can find via ohloh.net. If you can’t get Fortify, then do your best with FindBugs and CAT.NET, perhaps Pixy or others. The C language stuff is straightforward if you are using VisualStudio — tons of resources out there. But for others, it’s probably ideal to try the Hacking Exposed Linux 3E way of using either Klocwork or Coverity, depending on your industry’s code types. Embedded/aviation/etc should probably be Klocwork and Enterprise is usually Coverity with tons of rigor being applied to these. Also see the SATE analysis stuff, especially if you are hitting C/C++ hard.

If you don’t understand the code parts, you’ll have to go towards it slowly over time as static analysis isn’t the best approach if you already can pen-test stuff well from a full-knowledge system style test (blackbox is a total misnomer here). I suggest learning how to do server configuration traversals matching inputs (e.g. URLs and parameters) to the XML config file, to the jar/container/etc file, to the objects/source files, to the line of code [LOC]. Then it’s a matter of understanding how to read code (try the Code Reading book) and understand a good security approach (try TAOSSA Code Audit Strategies, Chapter 4). There are good books on debugging and unit testing that may also be worthwhile in this study.

Once you get to the LOC, you can either use a Fortify-in-IDE to trace the dataflow, or alternatively, just learn how to do this work using things like the OWASP code review guide, searching millions of books at once on books24x7 and safaribooksonline, etc. Maybe it’s best to work with developers. Sometimes the code is in a stored procedure in a database that makes an exec call or similar — so the code may not even be included in the buildable containers. There is a problem with static analysis tools and even manual code review that will lead to “lost sinks.”

Their practical advice for non-devs in code review: Don’t.

More and more, I think non-devs should skip over the code review, automated partially/fully or NO, and stick with understanding the framework (this assumes managed code). Managed code is awesome and the VMs themselves can be tweaked. Instead of just playing with Eclipse/Ant or VisualStudio/MSBuild warnings/errors/bug-finding-plugins — one can jump directly to the underlying framework and change the VM (i.e. harden the VM like one would harden a Cisco router or Linux box) using a tool like Reframeworker (see the Managed Code Rootkits book). So then, a security dood can type up all of the mistakes his/her developer-friends are making and tell the VM to spit out an exception, say, when the developer builds the code on a system with the hardened VM and uses, say, Statement instead of preparedStatement (and the other billion things to check for). This could potentially be a nice human-facing blacklist, much like the banned functions list that you can throw into VisualStudio.

Developers can work around these little intracacies, which changes the human factor up again. They can do lots of good and bad things. It’s good to know what they are doing, so you really have to be a developer in order to do this — or at least read their code and start to understand the psychology. Maybe you just need to interview them and learn a lot about liars and their lying (like that TV show with Tim Roth).

I like to take an architecture perspective because it’s totally high level and lead devs (who like to talk about this stuff) grok it well. This is again, usually only possible with OO and probably managed code languages. It  has to do with patterns.  Patterns are certainly something that netengs/sysadmins or whoever else can usually get, and if not, at least the basics.  One can dig pretty deep here. I have tons of ideas, but I’m not sure they will pan out. I think this is a huge area for excitement and improvement.

This leads me to threat-modeling discussion, which is really a horrible term. The BSI / Cigital is aligned with the old style OOAD, which I think has been ultimately replaced by DDT in the same way that CASE has been replaced by prototyping/refactoring, modern unit/component testing, and static analysis. They called it something silly like Software Architectural Risk Analysis.

Which brings me to BSSIM / OpenSAMM / Microsot SDL, etc. This stuff is trash.

Replace the above with actual real dev/QA improvements. I think appsec people should be adding appdev, apptest, and appperf value! It’s all about the trusted adviser role and teams, as before said over and over.

One might be surprised how often it is, in fact, not said over and over in the context where it needs to be said.  Instead what is often implemented is a hamster wheel of testing, detecting a few defects, fixing those low hanging fruit, certifying the resulting code product as “good enough” and then moving on without actually improving the process that generates flaws and usually imposing an ineffective and expensive bureaucratic cost to development.  I hear what he’s saying, but it’s my view that one needs some kind of process to encourage developer maturity programs.

I’ve seen a lot of manhours spent pushing secure development lifecycle efforts and development maturity frameworks to disapproving audiences.  Sadly, what I have seen happen more often than not is that it becomes not a developer maturity program that leads to fewer defects and a more efficient higher quality process, but a checkbox-driven recurring third-party code-audit, threat modeling, and pen testing cycle.  These can be useful, but only when performed at the right times in the development process and could easily be out performed by an approach similar to what they describe.

One conundrum that is really unsettled is the exploitation and risk management stuff. I have my own opinions, but they match Aspect Security very closely. This stuff also takes experience, but experience with knowing the business, business risk, data breach notification laws and how they play out, auditors for various compliance/regulatory crap and how that plays out, etc. It’s really whacky stuff that is so custom to any org.

I’ll be writing up a risk management thinktank churn piece sometime soon going into this (again) in greater detail.