StorageMojo

Chilling story about a security firm’s successful infiltration of a credit union’s infrastructure using old USB flash drives. They wrote a Trojan that would collect “. . . passwords, logins and machine-specific information from the user’s computer, and then email the findings back to us,” put it on the thumb drives and scattered them around the employee parking lot.

The bottom line:

Of the 20 USB drives we planted, 15 were found by employees, and all had been plugged into company computers. The data we obtained helped us to compromise additional systems, and the best part of the whole scheme was its convenience. We never broke a sweat. Everything that needed to happen did, and in a way it was completely transparent to the users, the network, and credit union management.

Of all the social engineering efforts we have performed over the years, I always had to worry about being caught, getting detained by the police, or not getting anything of value. The USB route is really the way to go. With the exception of possibly getting caught when seeding the facility, my chances of having a problem are reduced significantly.

Business Opportunity: Software or Epoxy?”
Do I sense a product opportunity? Software that erases everything on a flash drive that doesn’t have a security certificate? Or how else could one do it?

Or you could sell epoxy glue guns to seal off the USB ports. “Secure Goo” anyone?

Update
Back in 2002 I met several times with the CTO of a large defense contractor to discuss how my company could help them build a “network intrusion detection system”. He described a system that would take in about 5 TB of data daily from about 500 network monitoring points and load it into a flat file. That file would be massaged down to about 1 TB per day as unneeded datapoints were discarded. The SI on the job, who I also met with later, is the same one who’s done all the work on the discontinued TIA project and most of the other domestic spying jobs, Hicks and Associates. So I’ve concluded that “detection system” was in fact the NSA project, and that with 500 network monitoring points it is much larger than anyone in Congress knows about.

Also, it turns out that they aren’t limiting themselves to the scenario that Bruce Schneier so ably demolishes below: they are investigating people based on what they do, rather than who they know. Learn more at DefenseTech.org

This is getting weirder and weirder. Good for the storage industry until it stops though.
End Update

A recent article in Popular Science talks about the redoubtable Gordon Bell’s proposal for MyLifeBits, a machine that in just 15 years will allow you to capture, analyze, classify, store and search your entire life, like ILM, only it will actually work. Capturing video of every waking moment, listening to all your phone calls, archiving all the IM, mail, TV, calendars, meeting notes, every single boring banal moment caught in HD video and DTS 6.1 surround sound, transcribed, indexed, tagged and searchable. Bell, a famously forgetful and brilliant engineer who ran DEC’s R&D for 23 years, is quoted saying

Having a surrogate memory creates a freeing and secure feeling. It’s similar to having an assistant with perfect memory.

I’m not sure it would be a total panacea, if, like Bell, you habitually become so entranced with a brilliant new thought halfway through a sentence that you forget what you began to say. Instant replay, perhaps?

The relevance to the storage industry is, of course, that such devices would require terabytes of capacity every month. We could each have a Symmetrix mounted on a Segway tethered to our personal SenseCam following us about.

Or we could simply blast all the data wirelessly to a large, secure, data facility dedicated to keeping the data forever. Such as the National Security Agency, or NSA.

In an amazing bit of technical serendipity, while Bell is developing MyLifeBits to record all of our life’s data, the busy gnomes at the Intelligence Community’s Advanced Research Development Activity (ARDA) have a program called Novel Intelligence from Massive Data (NIMD). Novel intelligence refers to “actionable information not previously known”, a deliciously suggestive phrase for patriotic American’s proud of a knowing, action oriented government. As opposed to the ignorant and slothful government that mishandled Katrina, bungled the occupation of Iraq, and can’t balance the budget. You know, the one that exists outside the Intelligence Community.

NIMD is a data-mining program. And do they have data to mine. ARDA’s no longer available website stated

some intelligence data sources grow at the rate of four petabytes per month now, and the rate of growth is increasing.

Only four petabytes a month? Obviously they need more data. You can’t keep America safe with just 4 PB of data a month. And they need to store it. Forever.

Like Bell’s MyLifeBits, the NIMD data mining program is tackling the tough problems of data analysis and classification, looking at unstructured text, spoken text, audio, video, graphs, diagrams, images, maps, equations, chemical formulas, tables and so on.

Are they succeeding? They aren’t about to say.

Yet if you listen to the respected security expert Bruce Schneier, CTO of Counterpane Internet Security, it is an open question whether data mining can possibly succeed. As he noted in a Wired commentary

When it comes to terrorism, however, trillions of connections exist between people and events — things that the data-mining system will have to “look at” — and very few plots. This rarity makes even accurate identification systems useless.

Let’s look at some numbers. We’ll be optimistic — we’ll assume the system has a one in 100 false-positive rate (99 percent accurate), and a one in 1,000 false-negative rate (99.9 percent accurate). Assume 1 trillion possible indicators to sift through: that’s about 10 events — e-mails, phone calls, purchases, web destinations, whatever — per person in the United States per day. Also assume that 10 of them are actually terrorists plotting.

This unrealistically accurate system will generate 1 billion false alarms for every real terrorist plot it uncovers. Every day of every year, the police will have to investigate 27 million potential plots in order to find the one real terrorist plot per month. Raise that false-positive accuracy to an absurd 99.9999 percent and you’re still chasing 2,750 false alarms per day — but that will inevitably raise your false negatives, and you’re going to miss some of those 10 real plots.

So maybe it isn’t about data mining. Yet this expensively acquired technology can still help build a better, more secure America.

MyLifeBits, NIMD and MIT’s new Speechome project, where a professor is having his home wired for audio and video to create a 1 PB+ data store recording his child’s first nine months is certainly suggestive of a brave new world of full employment for storage professionals.

Mr. Bell, of course, thinks of MyLifeBits as a personal memory bank, a PDA for the ages. Yet once everyone is wearing one we can create an America where no crime goes unnoted — or unpunished.

We Can Still Monetize Our National Security Investment
Link all the surveillance cameras, implant RFID so people can be positively identified, capture the MyLifeBits data and run it all through the Son of NIMD, and we can have an America where everyone is responsible for their behavior 24 hours a day. No more security through obscurity or anonymity. Add GPS to automobiles and every incident of speeding can be tagged and fined.

Most crimes are misdemeanors, which means that mildly criminal behavior is easily monetized through fines. With lots of bahavioral data available we will be able to identify early warning signs of criminal behavior or psychiatric disorder. Perhaps we’ll even be able to implement the “pre-crime” units envisioned in the movie “Minority Report”, cops arriving just in time to stop the drug buy, the political bribe, or the underage actress drinking at a bar.

We’ll drastically reduce crime and reduce taxes as well. Isn’t that what we all want?

If you have nothing to hide, why would you object? Don’t you trust the government?

The Founding Father’s didn’t. Maybe we know something they didn’t. Or not.

Update
Fine post over at CIO.com blogs by Ben Worthen about a poor sapsucker whose name matches the name of some sleazoid criminal in another state. He can never get on an airplane without the 3rd degree; the DMV hassles him all the time. It really isn’t about trust. It’s about competence. As the biggest organization in our society, the government is only as good as we are. Sorry.

Jon William Toigo over DrunkenData.com has a post commenting on the Computerworld story on data exposure in online public documents. Using that data criminals can not only ruin your credit rating while collecting tens of thousands of dollars, they can also take your home. “A signature and notary seal extracted from an online county is all that is needed to take your home,” [David Bloys] said.” Scary stuff.

Bits just want to be free, despite the RIAA. The business value of electronically accessible data is just much greater than paper data. What we have is not a failure to communicate, but a failure to authenticate. If someone really can take your home with a signature and a notary seal, then the systems controlling real estate transactions are flat busted. Rather than try to plug every hole in every public and private entity that we share personal information with, why don’t we just make that information useless to criminals by controlling the systems they exploit?

As I’ve mentioned before (Data Security: A Modest Proposal) requiring authenticated personal approval for the release of credit reports would bring identity theft to a screeching halt. There are only three credit reporting companies. They all have lots of computers and smart people working for them. Authentication is a solvable problem. One law could choke this off tomorrow. Hiding every piece of personal information for 300 million people in millions of businesses and thousands of governmental units is neither likely or cost-effective — though the encryption folks might differ.

When Marshall McLuhan coined the term “global village” he was more right than he knew. In our digital global village privacy is no more likely than in any small town. As storage professionals we should tell our legislators the truth: storage technology cannot solve the problem of identity theft and fraud.

In the Middle Ages most people didn’t have last names. You were just the equivalent of Ed from LA, Bill from Memphis, or Leonard from Vinci. You didn’t travel far and everyone who needed to know who you were did. As people started traveling more a new infrastructure, including last names, was created to help identify individuals accurately.

Well, it is time for new infrastructure so we can securely authenticate ourselves both in person and online. It isn’t a storage problem and we’d be fools to take this monkey on our backs.

EMC’s GM of the Grid & Utility Computing, Ian Baird, mentioned at EMC World in Boston this week that EMC had invested in distributed caching technology developed by YottaYotta, a Canadian startup, for their “Grid Storage” strategic direction.

Distributed caching technology is crucial to creating WAN-based storage infrastructures that operate as if local, despite being spread over thousands of miles, where normal network latency would cripple response times. YottaYotta, an $80M startup based in Edmonton, Alberta, has been working on the technology since its founding in early 2000.

With Google’s gdrive initiative, as well as similar expected services from other major players, EMC is facing the threat of losing many petabytes of data now hosted on expensive EMC gear to web-based free or low-cost storage. As broadband adoption rates and quality improve, users will have less incentive to leave data in disaster-vulnerable single locations.

By promoting a network or grid based architecture to their lucrative corporate customers, EMC is hoping to stave off the fate of so many other big iron companies: being rendered irrelevant by high-volume, low-cost alternatives. This happened to minicomputers, 9″ and 5.25″ disks, mainframes, proprietary OS’s such as VMS, and countless other computer products. Storage hardware is one of the most profitable hardware businesses in the industry, helping keep otherwise money-losing computer hardware vendors afloat.

EMC, which is almost entirely a storage hardware and software vendor, except for their recent acquisition of VMWare, has been moving downmarket with lower cost Clariion products marketed by Dell and Intel. However the grid computing paradigm threatens to give big corporate customers a new and lower cost way to deliver storage services (such as IBM’s fledgling download grid) just as the web storage services like gdrive encourage downmarket customers to cut down on their storage consumption.

EMC’s biggest problem may be that grid architectures require greater integration between servers, networks and storage, integration that EMC can’t easily deliver since it has no server business. Maybe they’ll buy staggering Sun to get that piece.

What I don’t know and would like is whether EMC has an exclusive with YY. Distributed caching is really hard, and as other vendors realize they need it perhaps they will also go trotting off to scenic Edmonton (home of the free world’s biggest shopping mall) and lay their money down.

BTW, EMC spent $24,524M in 2005 and $20,297M in 2004 on strategic investments in private companies, which is the pot the YY money would have come from. No idea on how many other strategics they shared the money with.

I asked YY for comment, but no one was answering their phones. I hope to have more later.

Note: I was once employed by YottaYotta and still hold shares in the company.

Update:
A senior EMC technology manager would only confirm this morning that “We do have a grid project underway at an enduser site with YottaYotta, to leverage their caching know-how.” So it may be that no money has changed hands between EMC and YY. Investment contingent on successful trial? Or an NRE investment by EMC? In any case, congratulations to the engineering team at YY whose 6 long years of work on a very complex problem is finally bearing commercial fruit.

In this post Prof. Eugene Spafford of Purdue talks about security and passwords. There are so many buzzwords and scare-tactic marketing in security that most of the non-technical people I know are reduced to the cyber equivalent of burnt offerings: they subscribe to some (often) dysfunctional security product and then cross their fingers. They don’t have the time and energy to understand where the threats are and how to avoid them (like using Firefox instead of IE, for example). This article briefly delineates the major threat modes for passwords and then offers some thoughtful advice.

I saw an RFP recently for an organization spread over several hundred locations with thousands of employees that required the system to ensure that passwords were changed every month and that no passwords were reused over a seven years. There is no better way to ensure that thousands of passwords are stuck on post-it notes on monitors or under keyboards.

As the good professor concludes:

In summary, forcing periodic password changes given today’s resources is unlikely to significantly reduce the overall threat — unless the password is immediately changed after each use.

Giving users suggestions for selecting passwords that are both memorable to the user and difficult to research, and letting them keep them for a lengthier will result in better security and fewer lost passwords.

Even though disk storage gets about 5-10% cheaper every quarter, people still hate paying for it. A new CPU goes faster, a new display is brighter and/or bigger, but new storage just sits there until we fill it up.

For that reason, the idea of RAID 5 (see the World’s Shortest RAID Guide) seems to hold a hypnotic attraction for customers everywhere. While I understand that cheaper and almost as good is a win for most of us, RAID 5 is a mixed bag that may not do what you need, even if it does what you want.

Start RAID 5 definition
A formal engineering definition of RAID would require using words that I think many people would need defined as well, so I’m not going there. Operationally, a RAID 5 controller calculates data recovery information (parity) and spreads your data and the data recovery information across several disks, usually 4-10 disks. The big advantage of RAID 5 is that it protects your data while using only the capacity of one disk to do so. So if you have 6 400GB disks in a RAID 5 configuration, you have 2000 GB ( 6 * 400GB = 2400GB less the one 400GB disk of recovery info) of usable data storage capacity.

If you mirrored (RAID 1) those 6 400GB disks, you would only have 1200GB of usable capacity. Same disks, same power & space requirements, but 40% less capacity. For what?
End RAID 5 Definition

The technical answer to that last question is complicated, because it depends on what you are doing and how the RAID 5 is engineered. The non-technical (i.e. not for gearheads) answer is that by maintaining two complete copies of your data, RAID 1 (and its sibling RAID 1+0) will often complete individual reads faster, usually complete writes faster, and when a disk fails will protect your data better.

If there is a second disk failure in a RAID 5 disk group, ALL the data is LOST. Gone. Pff-f-f-t. So the natural question has always been: “How likely is a second disk failure?” Take the disk vendor’s MTBF (mean time between failure) data and posit a random distribution of disk failures, and the non-tech answer is: “not very.”

To illustrate, take a modern 400GB SATA drive with an MTBF spec of 400,000 hours. In a six drive RAID 5, like the one above, you would expect a drive failure once almost every 67,000 hours (400,000/6). Since there are only 8,760 hours in a non-leap year, that is about every 7.5 years. So no worries, eh?

Sorry, yes, there are worries, of two different types:

• First, what if the drive failures are not random? In my experience they frequently are not. Bad power, poor cooling, heavy duty cycles, shock and vibration problems, all come together to produce unexpected failure clusters. Even with a good environment, there will be clusters of failures simply as a function of statistical variation. So the random failure assumption is not always valid.
• Second, the problem of read failures. As this note in NetApp’s Dave’s Blog explains, complete disk failures are not the only issue. The other is when the drive is unable to read a chunk of data. The drive is working, but for some reason that chunk on the drive is unreadable (& yes, drives automatically try and try again). It may be an unimportant or even vacant chunk, but then again, it may not be. According to Dave’s calculations, if you have a four 400GB drive RAID 5 group, there is about a 10% chance that you will lose a chunk of data as the data is recovered onto the replacement drive. As Dave notes, even a 1% chance seems high.

Where Dave and I part company is in our response to this problem. Dave suggests insisting on something called RAID 6, which maintains TWO copies of the recovery data. Compared to our RAID 5 example above, this means that instead of having 2000GB of usable capacity, you would have 1600GB. And now RAID 1 would only have 25% less capacity. I say drop RAID 5 and 6 and go to RAID 1+0, which is both faster and more reliable.

RAID 5 and 6 use much more complicated software to create the recovery data in the first place, and then after a disk fails they need to read each of the remaining disks along with the recovery data to re-create the lost data. For large disks in large RAID groups this can take many hours, if not days. And while the recovery is underway your storage performance is hosed.

My point is, why even go there? Why not just maintain two complete copies of your data, so when a failure occurs, as it inevitably will (and at the worst possible time, of course) your data is just copied from one disk to another at disk-to-disk speed?

Small and medium businesses face enough uncertainty as it is. Spending a few extra bucks for RAID 1 or 1+0 will make your local digital data storage as bulletproof as it can be. Isn’t that what you really want?

The special interest attack on the Internet continues. Even as America falls further behind other industrialized countries in broadband penetration and network providers pursue plans to charge content providers for quality access, a new threat is emerging. According to an article in CNet, the Bush administration and some law enforcement agencies favor forcing Internet Service Providers (ISPs) to retain for lengthy periods, sometimes even years, their logs and other other data.

Why should SMB’s care? To the extent that the Internet enables SMB’s to broaden their market access, a free and healthy Internet is important. Unsurprisingly a number of powerful groups have found ways they could profit if the Internet was a little less free and healthy.

The latest is law enforcement. Under the guise of apprehending child pornographers a number of law enforcement officials are advocating data retention laws. Child pornography stinks and traffickers and consumers should be prosecuted to the full extent of the law.

What I have a problem with is a Congress that won’t lift a finger to protect consumers from the far more prevalent problem of identity fraud by asking three large companies to institute some simple procedures, while rushing to implement far more intrusive laws, affecting thousands of businesses and the confidence consumers have in this world-changing infrastructure.

Bad government is the single biggest evil in the world and has been throughout recorded history. The genius of the US constitution is that its system of checks and balances as well as its guarantee of individual rights limit the ability of the government to do evil. Erode those rights and weaken those checks and balances and you make it easier for the government to do evil.

I suspect the biggest problem with our child pornography enforcement is not the lack of ISP data, but the lack of funding. Let’s fix that problem first and then see what else is needed.

Slash dot points to this story from the BBC and AP about military secrets being sold along with USB drives in Afghan bazaars. The flash drives are stolen from US military bases by Afghans working in them.

A few thoughts:

• Maybe they should pay the Afghans a little more, and then fire a few for theft, to get folks to be concerned about keeping their jobs.
• Why the hell are sensitive files being kept on flash drives? Aren’t they are on reasonably secure network servers?
• This is exactly the same issue that US businesses face. Your entire customer contact list could walk out the door a dozen times a day and you’d never know until competitors are calling on them. This is the real data security problem, not social security numbers.

The seemingly insurmountable difficulty of protecting this kind of data suggests that we are ready for a re-thinking of how we store important data. Obviously the encryption/password model isn’t working. Or at least the current implementations aren’t.

DANGER! Marketer Trying to Design A Product

Perhaps some the additional metadata fields being added to advanced file systems could be used for a OS-based encryption engine. Save a new file and the dialog box asks, in addition to the usual stuff, if the file should be encrypted and who (owner, group) should be allowed to decrypt it.

Perhaps an admin level account could require that all files going out USB ports be encrypted. Or ??

This is a real problem. I’m not technical enough to design a solution, but it seems like the current processes are hopelessly broken. Any creative engineers out there with some ideas? This could be a very popular utility.

Computerworld has a story about Broward County, FL posting thousands of social security numbers and other personal data on-line, in compliance with state law. Personally I think it is a good idea that public records be available online.

What is a bad idea is making identity fraud easy, which is exactly what HR 3997, the Financial Data Protection Act of 2005, does. This bill does not allow credit freezes until you have already been a victim of credit fraud. How brain dead is that? Credit freezes are the single best tool against identity theft and Congress, at the behest of the banking industry, wants to outlaw them.

Please, write your Congressman and tell them in no uncertain terms that HR 3997 is unacceptable. America’s businesses are being saddled with billions of dollars in costs to protect against credit fraud when choking off the opportunity through credit freezes would be much cheaper and more effective.

Data security is a real problem, and a problem with multiple dimensions. The most troubling of these to me is the problem of identity theft, since it is a problem that hundreds of thousands of small and medium sized businesses should simply not be subjected to.

The basic problem is social security numbers. Many small businesses, especially in healthcare, have a need (in contravention of Federal law, IMHO, but wired into many systems nonetheless) for them, but a surprising range of businesses collect them on customers and employees. When they do the SSNs are typically accessible to a wide range of employees including low-wage clerical and temps.

With a social security number, an address and internet access, the enterprising identity thief can find birthdays, legal records and former addresses, allowing them to apply for credit and make purchases with only an estimated 1 in 700 chance of getting caught.

So millions of businesses are being told they need to protect social security numbers and other customer data to prevent identity theft and the potential for bad publicity and lawsuits. Vendors are happy to propose band-aid solutions that only cost money and time.

Don’t misunderstand: data security is a real problem. Yet protecting social security numbers from casual access shouldn’t be part of the problem. Why? Because the way people use SSNs for identity theft have one element in common: credit reporting agencies.

The issue is “identity fraud”, which to my mind is the real problem: we make it far too easy for thieves to open credit lines using stolen information.

Congress should require that all credit reporting services give out credit information only with express permission of the individual.

The credit reporting agencies will whine about this, since they are only interested in selling reports, not protecting consumers against fraud, or helping victims of identity theft. But since they’ve made the system so easy to game, I think they need to take responsibility for fixing it.

So what appears to be a “storage security” problem, in this admittedly limited case, is really a political problem. Congress, a wholly-owned subsidiary of the American Banking Association, is currently considering a Federal law that would pre-empt state laws that allow consumers to require persmission before releasing credit reports. This move takes a responsibility that could easily and at no cost to taxpayers be placed on three national firms and would allow consumers to freeze their credit reports only after they’ve been victimized!

I encourage all small businesses that like the freedom to use SSNs for business purposes to contact their congressmen to protest this stupidity. Rather than tax millions of small and medium businesses with the responsibility to protect (or not use) SSN’s, let’s give it to the three large and well-financed reporting agencies, where it belongs.

Well, after getting /.’d yesterday this is bound to be an anti-climax. But I scoured the exhibition show floor the coolest products at SNW last night and found three contenders.

The first is Wasabi Systems the maker of a small (~1″x 3″) flash drive loaded with software that plugs into a motherboard’s IDE port and turns an ordinary pizza box server into either a NAS device, an IP-SAN device or both, with a software RAID module as an option. This all runs on Wasabi’s proprietary version of BSD, called Wasabi Certified BSD (please, let’s not get into a flame war about BSD vs Linux or GPL vs BSD licensing - I’m not endorsing or dissing either — just reporting). What’s cool is that in a few minutes a standard server (get Wasabi’s support list here) can become a NAS or iSCSI device. This is a natural for the SMB market where cost-effectiveness, ease of use and implementation speed are critical. It looks like DataCore’s SanSymphony product done right, i.e. on a stable Unix base.

Next on the list a company called Index Engines that “indexes data at wire speeds” or at “up to” 2Gbit/s all the data from your backup software (NetBackup, NetWorker, Tivoli Storage Manager) as it goes off to disk or tape. The indices are about 8% the size of the original data, so a server with 1TB of storage can index about 12TB of data. They claim to do full content indexing of more than 80 different document types, including pdf, doc, xls, pst and Exchange. Plus you can cluster up to 64 Index Engines together “providing unified search results for over 4 billion documents” (do I sense a 32-bit document address space here?). You access a Google-like interface through a browser to search.

While I am not crazy about the apparent requirement to index the data as it is backed up, I do believe this concept will help hasten the long overdue death of hierarchical file systems and the folder metaphor they support. Search and metadata extensions (such as IE’s indices) aren’t just the coming thing, they are the only thing, IMHO.

And finally, partly on the basis of the best and most extensible elevator pitch I’ve heard in a long time, Availl’s Wide Area File System. They claim that “No matter where users are, or how many open the same file at the same time, Availl ensures that only one user has read/write control.” Sounds like Einstein’s “spooky action at a distance” take on entangled quantum particles, but they stress that their byte-level differencing technology is what keeps bandwidth requirements to a minimum and latency low. All data is stored locally, presumably only the lock/unlock commands and byte differences get communicated, so data access is very fast. They mirror all the data in the WAFS across all the sites, so users can operate even when the network is down.

As to which is the coolest — well, you can decide for yourself. I like the Wasabi product best and plan to learn more about it. My biggest question: if it is as easy to use and configure as they say, why not sell a version directly to end-users to plug into already owned servers?