Amazon downtime

There was recent news about how Amazon was down for two hours. Speculation runs rampant on cnet about the cause:

“It doesn’t seem to be the result of a network-initiated attack, at least from my preliminary analysis from our probes,” Ranjan said.

Human error may not sound as gripping a tale as a network attack, but there’s plenty of drama for the people responsible. And it’s the career-limiting variety of drama, said Illuminata analyst Gordon Haff, who hazarded a guess that Amazon’s problem involved its front-end Web servers.

The security group of WebSense, a Web site and communications protection company, also saw no evidence Amazon’s problem was security related.

Having talked to a lot of Amazon people here after my arrival in Seattle, I’m surprised that they don’t have more downtime. Amazon is run like a huge basement operation.

Let me explain.

Amazon doesn’t have a real operational staff. They have developers that code up releases by day and then have to handle first-line response to outages and incidents by night.

As far as I can tell, they have no industry standard monitoring software, configuration management platform, or even any centralized policy framework. They leave everything up to business units to develop all of their own infrastructure and systems management strategy. Best yet, it’s all run by developers.

I think everyone reading this who has been a pro in running operational systems just recoiled in horror after that last sentence.

I understand that entrepreneurial environments want to be as nonconforming and iconoclastic as possible as to “think outside the box” or whatever in-your-face-status-quo stance to encourage innovation, but don’t take that kool-aid to the harsh realm of uptime.

Stability in operational systems by standardizing their build process, quality assurance of code deployments, and operational staffing that doesn’t tax your architectural staff not only leads to better performance, but it also takes your staff out from under the Sword of Damocles of downtime. Having to choose between stability and innovation is a poor choice to make when you can have both, and a cost savings, with a bit of operational sanity.

The encrypted traveler

As border enforcement as using increasingly invasive tactics, a traveler that has any privacy concerns for the data that they are carrying (especially if visiting the United States) will very likely take steps to protect themselves.

Examples:

FindLaw:

The Ninth Circuit, in a decision announced this summer, has approved forensic searches of laptop computers at the border, even when the laptop’s owner spent no time outside the airport in the foreign country and was under no suspicion of possessing foreign contraband.

Washington Post:

Nabila Mango, a therapist and a U.S. citizen who has lived in the country since 1965, had just flown in from Jordan last December when, she said, she was detained at customs and her cellphone was taken from her purse. Her daughter, waiting outside San Francisco International Airport, tried repeatedly to call her during the hour and a half she was questioned. But after her phone was returned, Mango saw that records of her daughter’s calls had been erased.

A few months earlier in the same airport, a tech engineer returning from a business trip to London objected when a federal agent asked him to type his password into his laptop computer. “This laptop doesn’t belong to me,” he remembers protesting. “It belongs to my company.” Eventually, he agreed to log on and stood by as the officer copied the Web sites he had visited, said the engineer, a U.S. citizen who spoke on the condition of anonymity for fear of calling attention to himself.

Police Blotter:

What: A business traveler protests the warrantless search and seizure of his laptop by Homeland Security at the U.S.-Canada border.

When: 9th Circuit Court of Appeals rules on July 24.

Outcome: Three-judge panel unanimously says that border police may conduct random searches of laptops without search warrants or probable cause. These searches can include seizing the laptop and subjecting it to extensive forensic analysis.

Ars Technica:

Stuart Romm boarded a plane in Las Vegas on February 1, 2004. When he got off the plane in British Columbia, Canada’s Border Services Agency stopped Romm for questioning. After learning that Romm had a criminal background, Agent Keith Brown searched his laptop and discovered child porn sites in Romm’s Internet history list. Canada then bundled Romm back onto a plane to Seattle, where US Customs agents had a chance to question him further.

They also conducted a forensic scan of his hard drive and turned up images of child pornography in Romm’s browser cache. The images had been deleted (intentionally, it appears), but were recovered by an agent using software called “EnCase.” Romm then admitted to investigators that he used Google to search for child pornography, and that his “therapy” had failed to help him quit.

Why is it always the pedophile that is used as an example of why invasive measures are justified? Perhaps all civil liberties should be put to the pedotest.

Pedobear_13.png

Because of the perceived need for such methods in several countries, many people, including business travelers with trade secrets, choose not to travel with any data on their person at all and access their data online when they have reached their destination.

Toward this end, I would like to call to mention this excellent document produced by the ninjas who make TrueCrypt. The concept of the hidden service via tor or the hidden volume via TrueCrypt will become more and more popular as long as searches and information harvesting becomes increasingly aggressive.

Why I hate BlackBerries

I have been working hard to avoid Blackberries of all kinds having seen sales people (who if you ask anyone who works with technology, they will tell you that people in sales push for the worst solutions available almost all the time) fiddle with them for years.

  • They never quite worked right.
  • Their voice quality sucked.
  • They’re a closed platform.
  • The integrate with Exchange as some kind of parasitic add-on module (as if running a Windows mail server wasn’t enough of a threat exposure)

Clearly, theirs is like the ultimate recipe for suck.

So I avoided them. I would say things like “You have a business case for me to have mobile email? No problem. I’ll take care of it.” I would then have some kind of mail solution of my own that would work well, integrate with everything else I was doing, and not drive me insane.

Before I complain any more, I will give it up for one thing that Blackberry does do. They push a mobile security policy to their devices that can involve remotely wiping the handheld

They really can’t take credit for all of this as everyone else supports it as well, but it’s a good thing from a governance/management angle. It is obvious that they would need it first because of their sales-centric user base, but necessity is the mother of invention. It’s also the mother of horrible duct-tape-style nasty rigging of solutions.

After dorking around with one of these consumer-level Blackberries and noting how it would ring occasionally and just vibrate at other times. It would perform randomly when I expected things to work all of the time. Additionally, their touch-typing is primitive when compared to other phones. It did not please me.

Enough of this. Can I use my old Nokia e61? It has blackberry software. Shouldn’t it work?

Apparently not. I gave it a good try, but there would be some version incompatibility or hidden password (likely inserted by carriers) that would prevent me from using the software successfully.

This really isn’t surprising why this might be if you look at the Nokia BlackBerry Connect page and look at the completely different dependencies for each of the carriers. If you’ve upgraded your firmware, as I’ve mentioned before is always a good idea, then you can’t use BlackBerry software with it. If it’s supported at all. If you look at BlackBerry’s own site, you get a huge list of carrier sites where you might be able to download a specific supported out of date build.

So let us consider this a moment and ignore some of the exceptional cases. This, usually, is a service that pushes email from a service that a business owns to a handset that a business owns transported over a cellular network.

So why all the dependency and pitfalls for using software? Is it the case that cellular providers believe that handsets should never be touched by end users or even corporate customers and if you do, to fix a vulnerability for instance, they just shouldn’t work anymore.

Having to choose between functionality and security is not fair.

I suppose it makes some sense that they don’t want to support their software on other smartphones as they would prefer you purchased their handset platform as well, but what about supporting people who purchased their enterprise products? Is the message “Too bad, buy more of our stuff”?

Backward, trouble to manage, and poorly performing. I guess I’ll continue to be surprised that people continue to use them. It really shouldn’t be a surprise to anyone that Android and iPhone are going to dominate the market in the next couple of years.

It is a question of usability.

Does this industry really intend that users need to continue to decide between functionality and secure operation? Why isn’t this seen as completely ridiculous? There isn’t any value in requiring a middleman between enterprise software and the platform where the client software runs.

AT&T hates their customers

Every time I have an interaction with AT&T wireless, it is an agonizing and drawn out horror of an experience. Because I know this, I only call then when absolutely necessary. Basically this is when they break things and I need to figure out why my stuff is busted.

I spent about four hours on the phone with AT&T after my wireless data was mysteriously half-broken. When someone decided that they were pissed off or didn’t want to be helpful, I just hung up and called back in again. There really isn’t any point in taking up any more of my time in

After one of their higher-level techs spilled the beans that AT&T has implemented some new program of removing functionality that customers are paying for based on the IMEI of the phone assigned in the customer account.

Why should you care? I thought that it was interesting that I was no longer getting the service I was paying extra to make sure I received. I spoke to another rep in business sales (another good trick to get decent service is to go through business services as normal customer service is always pissed off, semi-literate, don’t care, or a combination thereof) and he said that he had lost data service on his blackberry about a week ago and that he was likely having the same problem.

Through the course of my research of trying to figure out what they screwed up so that I could tell them how to fix it (this is the only way to handle any telco, by the way), I found several other interesting tidbits.

I took a bit of a longer view of how AT&T manages their customers and their service agreements in order to be prepared for my encounter. Ever since data plans have been offered, consumers have been using the abilities that were built into the phones for this purpose to attach tablets, laptops, and other peripherals to the data service on their phones. This hasn’t been a very big deal until recently and, much like SMS was before it became popular, it was largely free as it was not commonly used by the average consumer.

Now that it has, it is worthwhile to take note of some of the strange language in the agreements for their “unlimited data” plans, which aren’t so unlimited:

DATACONNECT PLANS
DataConnect plans may ONLY be used with AT&T-certified LaptopConnect (PC Data) Cards and eligible AT&T-certified customer owned and maintained (COAM) devices for the following purposes: (i) Internet browsing; (ii) email; and (iii) intranet access (including access to corporate intranets, email, and individual productivity applications like customer relationship management, sales force, and field service automation). The parties agree that AT&T has the right to impose additional charges if you use more than 5 gigabytes in a month. Prior to the imposition of any additional charges, AT&T shall provide you with notice and you shall have the right to terminate your service.

PDA/BLACKBERRY PLANS WITH TETHERING
PDA/BlackBerry plans with Tethering may ONLY be used with AT&T-certified RIM BlackBerry devices and PDAs for the following purposes: (i) Internet browsing; (ii) email; and (iii) intranet access (including access to corporate intranets, email, and individual productivity applications like customer relationship management, sales force, and field service automation). PDA/BlackBerry plans with Tethering may be used to tether such PDA and BlackBerry devices to a Personal Computer. The parties agree that AT&T has the right to impose additional charges if you use more than 5 GB in a month. Prior to the imposition of any additional charges, AT&T shall provide you with notice and you shall have the right to terminate your service.

Source from the AT&T Wireless Terms of Service.

The bold is theirs.

Apparently this is enforced rarely and only as a hammer to punish customers that piss them off.

As you might expect, this has been found and reported a couple of times and usually at Howard’s Forums and reported here by dslreports.

‘Unlimited’ AT&T Wireless Data Plans About To Be Capped?
Rumblings among insiders about implementing 5GB quiet cap, like Verizon…
09:33AM Friday Jan 11 2008 by Karl
tags: prices · business · wireless · bandwidth · Cingular Wireless
An anonymous AT&T insider yesterday hinted to us that the company’s wireless division would soon be implementing a 5GB monthly usage cap on some unlimited data plans. We contacted AT&T for official comment and were told that there’s no changes in store that they’re aware of, but they’d nudge us if anything official came along. Today we’re seeing some discussion over at Howard’s forums that would seemingly confirm there’s some changes coming:

Click for full size

I just heard that the pda plans will no longer be unlimited but will be capped at 5 gigs. Users will not be billed overages but people with constant overages will be contacted to try to reassess the users needs. The new plans are nationwide so I’m not going to disclose my market but they are getting rid of the media bundles and M2M messaging. Text and data is now separate. PDA plans will be lowered to 30 bucks to match blackberry personal and media net unlimited is lowered to $15 bucks.According to the poster, the plans will be live in a few weeks. Assuming these looming changes are true, AT&T may want to start removing the word unlimited from their advertising material. Verizon, who similarly advertised an unlimited service that actually had a 5GB monthly data cap, was busted last October by the NY Attorney General for false advertising. When we hear more on these rate changes we’ll let you know.

..and reported again a couple of months later:

AT&T’s 5GB Wireless Broadband Mystery Cap
Heavy users can prepare to pay a fortune…
03:13PM Tuesday Mar 18 2008 by Karl
tags: business · wireless · bandwidth · Cingular Wireless
For years, Verizon Wireless was trying to have their cake and eat it too, by advertising their EVDO service as unlimited, but quietly imposing a 5GB monthly cap. That advertising charade ended courtesy of NY’s attorney general (no, not client-9) last fall. Back in January, AT&T insiders insisted that the company was preparing to apply a 5GB monthly cap of their own to their unlimited HSDPA service. The company’s terms of service already states as much:The parties agree that AT&T has the right to limit throughput or amount of data transferred and/or deny, disconnect, modify and/or terminate Service if you use more than 5 gigabytes in a month. If you require more than 5 gigabytes per month, ask us about our DataConnect 5GB Overage plan.That plan doesn’t appear anywhere on AT&T’s website. Gearlog called in to ask about the plan, and found that actually using AT&T’s wireless broadband network in any volume can be a very pricey proposition:if you call in, you’ll find it’s $350/month for 5GB, plus $0.50 per megabyte (really, $.0005 per kb, but my megabyte formula is more readable.) Since you’re probably a heavy downloader, let’s think of that as $500 per gigabyte. Yes. They want to charge you $350 for exactly what you’re paying $60 for. Want 10GB instead of 5GB? That’ll cost you $2,850 for that month. Now, to be fair, an AT&T rep told me that they’ll probably give you a pass for a month or two if you accidentally go over 5GB. Then they’ll give you a call and try to convince you to move to the Punitively Expensive Plan.AT&T’s website still advertises “unlimited” data for Blackberry and PDAs provided you don’t tether, but we’d be interested to see if any users have tested the boundaries with smart phone consumption alone.

Note that life as a bandwidth hog on Verizon’s network is no easier. Buried amidst all the fawning adoration of Verizon Wireless for their recently announced unlimited yammering plan was the fact that they implemented some very pricey data overage charges of their own.

I had thought that perhaps it was possible they were mad at me for actually using the service I was paying for, but there was no way I was even using a significant fraction of 5G down a month.

So I looked on Howard Forums and saw that a whole bunch of people were having the same problem. I contributed a bit to these threads with the information I gathered from my hours of talking to AT&T representatives, and then focused on fixing my problem.

After this, and several other misadventures, I don’t suggest anyone even talk to customer care. Just don’t bother. It is a waste of time. If you want to do it at all, do it to make AT&T lose money. It costs money to staff call centers though, as it turns out, AT&T will soon be charging you to speak to a human. It really is amazing how minimum requirements of business are becoming features that one gets billed for using.

So after all this, I had a surprisingly simple solution to my problem.

I went to an AT&T storefront. I told them that business customer care told me that I needed a new SIM. I asked them to delete my data plan and the phone that was currently in my account. They scanned my IMEI number from the back panel of my phone, added a data service to my account, scanned in the new SIM card and gave it to me.

All was fixed and operational again after a few minutes.

The point of this is that cable and telco companies, who have re-established their monopolies, are increasingly using business practices that lock in customers instead of satisfying them. Since the consumers have no choice, the are billed a regulated amount as set by the government.

This is the case because much of this infrastructure was built with your tax dollars. Many of these companies have benefited from this but conveniently forget about this when they ask for less regulation because “they built the network and now it is theirs to do what they want” as is the argument for many companies net neutrality defiant behavior.

So do what needs to be done to defeat the problems generated by companies that don’t really care if they break services that you have come to rely on to do your business. Take the path of least resistance and remember that you owe them no consumer loyalty, because they do not appreciate or respect your patronage.

My talk at Seattle Toorcon 2008

I gave a little talk this weekend at the second Seattle Toorcon.

My presentation is as follows, though as usual, I ad lib when presenting. Video may appear in the future.

Read More »

iphone meets net neutrality

Chris Soghoian made an interesting point in his recent cnet article entitled iPhone rules pose Net neutrality, antitrust concerns. My key point of interest here is the following:

Apple’s, and to a degree AT&T, inclination to lock in customers is counter to the current direction of the market. Verizon is opening their cellular network and allowing Amazon to sell EVDO data products bundled with their Kindle. Google is backing the Android open platform handset. Most popular websites have opened their API.

Most business progress in recent times has been to open platforms to applications, unconventional business opportunities, and making their customers happy by giving them additional freedom of choice. Apple and AT&T defies this path at their peril.

Some analysts think around 1 million of those iPhones have been purchased with the intention of unlocking them to run on other cell networks. If those numbers are true, that means iPhone unlocking exploded in the fourth quarter despite two steps taken by Apple to reduce the number of iPhones bought with unlocking in mind.

Tom Krazit - “iPhone unlocking explodes despite Apple’s countermeasures

As shown by statistics such as the gap in units sold to those activated, it is clear that the technophile population is quite comfortable in guerrilla tactics to achieve the functionality they want.

Companies that wish to be, or continue to be, successful should keep this fact in mind.

New Facebook private features

Facebook has a worthwhile privacy improvement today. It’s nice to see social networks giving some configurability of granular access to the information that users make available there.

One thing that is not obvious is how to block limited users from information that you would rather not share for whatever reason. The “Limited Profile” blocking is non-intuitive, so I thought that I would outline the process here:

Picture 8.png

As you can see, you need to type in “Limited” and select it when it pops up. It then appears like so:

Picture 9.png

Intensive Smash Bros Training

As I never bothered to buy a Gamecube, I only played smash brothers with drunk people at college parties and when I visited my cousins a bunch of years ago.

With Super Smash Brothers Brawl now out for Wii, my lack of skill is showing and must be corrected to preserve my nerdly reputation.

What is there to do to be ready for the challenges that will be thrown my way over internet play? Only one thing: Intensive Smash Training.

Fortunately for me, I have been given some small clue from reading the n00b instructions here. However, I don’t think there will be any substitute for being destroyed repeatedly.

My ID for those wishing to humble or team up with me is: 4382-1723-3425

Gordon Ramsay: Food Hacker

I really enjoy cooking. I like it because eating is a fundamental part of every human life that everyone can appreciate inherently. I also because there is no “correct method” of doing it. There are nearly unlimited variations of preparation, ingredients, and palates.

With all of these variables, I think most people would agree that cooking is more of an art due to the amount of variables and interpretation involved. It is controlled chaos, but at the same time, a knowledgeable foodie can agree that some dishes are empirically better than others.

In some of my down time, I’ve been watching Gordon Ramsay on the BBC. There can be little doubt that he is a ninja in his own respectable and creative way. The way he addresses and attempts to deal with problems as a consulting chef and restauranteur is very much a hackers approach; a deep rich knowledge and experience with a passion for the art and finding creative and tailored solutions to problems.

All the while he speaks in a blunt effective manner tempered with occasional diplomacy. This is a very alpha behavioral modification method.

  • Efficient
  • Effective
  • Strong
  • Takes no crap, yet open to argument and new ideas

Very hacker mentality. If you replaced the bollocks with pwned in his Kitchen Nightmare conversations it might remind you of some other conversations of the nerdkind chats you have heard in the past.

I used to enjoy Alton Brown, but not anymore. What was appealing was the clear chemistry and sort of high school science class approach to cooking. What became tired was that the guy is a slob and has no talent for the art of the experience. Having watched an episode of the lame American version of Iron Chef may have had a lot to do with thinking less of him. He has kind of the same approach as presented in the Cooking for Engineers website with Southern cultural references.

Really what I’m going for here is that to a large degree, technology proficiency is much the same as chef proficiency in that many of the goals and experience is comparable. Some pros have formal training and others have followed more of a vocational path in working their way up through the ranks. In my experience, what really makes someone a standout is passion and deep experience and enjoyment of the art. Where talent and passion is understood in the culinary industry, as it is likely to be the second-oldest profession and traditions are deeply established, most standard industry environments do not appreciate the nontraditional path of the technology artist.

I would think that all would be well served to appreciate these similarities.

PGP key replacement time

I have a new pgp key.

Feel free to sign it at your pgp.net keyserver of choice.

$ gpg –fingerprint FE264BAA

pub 16384R/FE264BAA 2008-01-17 Key fingerprint = 9A05 99DB 838D 9049 0509 AD5D 26A4 2F8A FE26 4BAAsub 16384R/3CA219BB 2008-01-17

Why so big a keysize? A friend of mine cited the following:

NIST key management guidelines further suggest that 15360-bit RSA keys are equivalent in strength to 256-bit symmetric keys

keysize.png

NIST gives the following requirements. Other cyphersystems can be compared on the same site.

If something is worth encrypting, why not do it seriously and stay ahead of innovation.

At this time, gnupg does not allow keys of this size. To be able to generate one, simply remove the limitations from the source, compile as usual, and generate your keys. George Hill has a patch file here that you could use as a reference.

--- gnupg-1.4.7/g10/keygen.c.orig Fri Dec 21 21:33:27 2007 +++ gnupg-1.4.7/g10/keygen.c Fri Dec 21 21:34:51 2007 @@ -1502,12 +1502,12 @@ static unsigned ask_keysize( int algo ) { - unsigned nbits,min,def=2048,max=4096; + unsigned nbits,min,def=2048,max=16384; if(opt.expert) min=512; else - min=1024; + min=2048; switch(algo) { @@ -1525,7 +1525,7 @@ break; case PUBKEY_ALGO_RSA: - min=1024; + min=2048; break; } --- gnupg-1.4.7/util/secmem.c.orig Mon Feb 12 06:26:30 2007 +++ gnupg-1.4.7/util/secmem.c Fri Dec 21 21:44:55 2007 @@ -58,7 +58,7 @@ #endif -#define DEFAULT_POOLSIZE 16384 +#define DEFAULT_POOLSIZE 131072 typedef struct memblock_struct MEMBLOCK; struct memblock_struct {