In a shock move, Microsoft today announced that it had abandoned development of Windows Home Server’s Drive Extender storage technology. The announcement comes almost eight months into a public beta of the next version of Windows Home Server, codenamed “Vail”, and will result in Vail, Windows Small Business Server 2011 Essentials (Codename “Aurora”) and Windows Storage Server 2008 R2 Essentials (Codename “Breckenridge”) shipping without Microsoft’s advanced storage subsystem.
A blog post by Microsoft’s Michael Leworthy outlines today’s decision:
During our current testing period for our SBS 2011 Essentials and Windows Home Server code name “Vail” products, we have received feedback from partners and customers about how they use storage today and how they plan to use it moving forward. Today large hard drives of over 1TB are reasonably priced, and freely available. We are also seeing further expansion of hard drive sizes at a fast rate, where 2Tb drives and more are becoming easy accessible to small businesses.
When weighing up the future direction storage in the consumer and SMB market, the team felt the Drive Extender technology was not meeting our customer needs. Customers also told us that they wanted easier access to data stored on Drive Extender drives so they are able to view these files outside of Drive Extender. Therefore, moving forward we have decided to remove the Drive Extender technology from Windows Home Server Code Name “Vail” (and Windows Small Business Server 2011 Essentials and Windows Storage Server 2008 R2 Essentials) which are currently in beta.
Drive Extender, seen by many as one of Microsoft’s most innovative engineering feats of recent years, was a storage replication system that managed Windows Home Server’s storage pool, allowing the use of any combination of internal and external drive types, removing the need for drive letters and providing duplication of files and folders to protect from hard disk failure. It has not been without its fair share of controversy, however – a serious bug in the technology was identified in October 2007 which caused data corruption in certain circumstances. It took Microsoft two months to acknowledge the seriousness of the issue, and a further seven months to fix the bug.
Back in August 2008, Charlie Kindel, then General Manager for Windows Home Server at Microsoft outlined the guiding principles of Drive Extender, the spirit of which runs right across the platform “as a server designed for ordinary people”:
Windows Home Server storage system design requirements
- Must be extremely simple to use. Must not add any new concepts or terminology average consumers would not understand. Simple operations should be simple and there should not be any complex operations.
- Must be infinitely & transparently extendible. Users should be able to just plug in more hard drives and the amount of storage available should just grow accordingly. There should be no arbitrary limits to the kinds of hard drives used. Users should be able to plug in any number of drives. Different brands, sizes, and technologies should be able to be mixed without the user having to worry about details.
- All storage must be accessible using a single namespace. In other words, no drive letters. Drive letters are a 1970′s anachronism and must be squashed out of existence!
- The storage namespace must be prescriptive. In other words, our research told us that consumers want guidance on where to store stuff. Our storage system needs to be able to tell users where photos go. Where music goes. Etc…
- Must be redundant & reliable. There are two components in every modern computer that are guaranteed to fail: fans and hard drives. Because they have moving parts, Windows Home Server must be resilient to the failure of one or more hard drives.
- Must be compatible. Compatible with existing software, devices, disk drives, etc…
- Must have great performance.
- Must be secure.
- Must enable future innovation. Both the amount of storage consumers are using, and capacity/$ are growing at Moore’s Law like rates (while nothing else really is). This creates a discontinuity in the industry and an opportunity for innovation. The storage system must operate at a higher level of abstraction to enable rich software innovation (file level vs. block level).
Following Kindel’s departure from Windows Server organisation(to a new role in the Windows Phone 7 team), the clarity (and bravery) of DE’s design principles were compromised early in Vail’s development, admittedly with a number of resulting technical benefits. DEv2 offered greater compatibility for applications that needed to directly interact with the drive pool, created the ability for Windows Home Server to duplicate and move data when files were open (removing the file conflicts often experienced in v1), and improved performance by the removal of drive balancing. But those much needed improvements came with their own technical challenges and led to dilution of the core principles highlighted by Kindel.
Drive letters, that 1970’s anachronism, crept into the platform for the first time. Unlike WHS v1 which presented shared folders in a single volume, in Vail releases to date, shared folders became dedicated volumes with their own drive letter.
With limitations on shared folder allocation as a consequence (there are only 26 letters in the alphabet), pre-configured personal folders for each of the ten supported Windows Home Server users were cut from Vail, further compromising DE’s guiding principles. Whilst Microsoft have claimed that personal folders were rarely used, the change conveniently removed the need to reserve those drive letters for user accounts. Even more seriously, attempts to re-engineer Drive Extender for Vail led to significant limitations in the number of drives that could be supported with stability during the beta, and the move to storing data across multiple drives in 1GB chunks led to two major headaches. Firstly, inefficiencies in the release of free space back to the storage pool from data deleted by users led to Microsoft’s Home and Small Business Server team needing to call in defrag specialists Diskeeper and Raxco for third-party support. Secondly, with data now striped across multiple disks, Microsoft created an inherent weakness in DEv2 whereby a single disk failure could take out a significant number of files, depending on how many files had parts stored on the failed drive. Most controversially, in WHS v1, in the event of a system drive failure, files could be recovered by simply plugging a storage drive into another PC and copying the data from the drive. DE v2’s switch to a custom, block-based file system did not allow this ease of recovery.
Over on Microsoft’s Windows Home Server support forums, DE v2 has been subjected to continued criticism from beta testers since the release of the Windows Home Server “Vail” beta in April:
DEv2 uses technology that is as far as I know present in no other Windows distribution, seems more unreliable (as it uses data striping which decreases reliability) and uses a quite heavy additional overhead.
I am using WHS because of the Drive Extender, it [is] a unique solution that allows me to gather up all the junk that used to be spread all over my home network on different shares and box it all in one easily accessible accessible location. Now it looks to me like Microsoft is ignoring their current unique usefulness and going over to the mainstream backup and safety data storage. This is all good, but its also something that you can get from 100 different vendors.
WHS1 with its ” just a bunch of disks”, works just fine however, WHS2 does not, as any disk that has failed under this system can’t be read by any other readily available system that’s “to hand”, for the average home user, to me this is a fatal design flaw.
With WHSv2, DE has now dropped, to all intents and purposes, to RAID-0 levels. ANY single drive, out of my 12 failing, and I lose ALL my data, at least for the aforementioned larger files. Bottom line is this is a lousy decision..and hopefully one that can be reversed.
