When the topic turned to cloud storage, I think we all agreed that automation would be key.  I made the comment that we humans are all fundamentally lazy and anytime we have a choice between man or machine to do mundane/difficult work we always opt for the machine.  In the cloud, this means that policy-based storage provisioning and data protection will become paramount.  Greg asked whether or not I thought this was the right direction to be going.  I responded that we really had no choice, the sheer volume of data we will soon be asked to manage will simply be impossible for humans to manage, so we’ll be handing control over to machines, like it or not.

We covered many other topics, such as examples of great successes and failures in the history of the data storage industry, and I gave a sneak peek into my next planned book “Building the Virtual Mainframe.“  All and all I thoroughly enjoyed being part of this podcast (yes even the banter) and I hope to participate in more Infosmack’s in the future…

In the meantime, here is a complete list of all the books authored by the people on this podcast:

Building Storage Networks Marc Farley

Storage Networking Fundmentals Marc Farley

Evolution of the Storage Brain Larry Freeman

The Insider’s Guide to Data Deduplication Larry Freeman

Backup and Recovery W. Curtis Preston

Using SAN and NAS W. Curis Preston

Unix Backup and Recovery W. Curtis Preston

Enjoy,

Larry

1. Elastic, quickly adapting underlying infrastructure to changing subscriber demands
2. SLA-driven, automated and integrated to provide swift response times
3. Policy-based, with deep levels of automation to move data as required
4. Secure and reliable
5. Able to control geographically dispersed data

Now let’s take the points above and simplify them a little more to describe cloud-enabled storage:

1. Cloud-enabled storage is virtual – data is not bound by physical arrays or geography
2. Cloud-enabled storage is elastic – storage resources can be assigned transparently
3. Cloud-enabled storage is automated – pre-defined policies dictate the behavior of data
4. Cloud-enabled storage is secure – tenants share each other’s resources but not their data 

I’ll come back to those 4 points in later blogs, but in this blog I want to discuss a problem that emerges when you have a virtual, elastic, automated, and secure IT infrastructure.  That problem: how do you figure out who pays for the storage services you are providing?

In the old days, a project manager would go IT and said “I need a couple servers and 10TBs of storage for my new application.”  You’d pull out your vendor pricelists and add up all the costs (including network switches, backup software/hardware and of course your valuable time) and tell this manager, “OK I can have that ready for you in a couple months and I’ll need to charge your department around 100 thousand dollars.”

Once you both agreed to this amount, you’d go off and build out this project and when the vendor bills arrived, you’d charge the costs to the appropriate department.

In the era of cloud computing, however, this model changed.  Instead of buying servers, you simply create a couple of Virtual Machines.  Instead of buying a storage array, you carve some storage capacity out of your internal cloud and assign a backup policy.  Viola! While your friendly project manager is still standing there with his/her mouth agape, you say “OK that should do it, anything else?”

That’s the beauty of cloud computing and cloud-enabled storage – you can quickly and efficiently serve the IT needs of your organization.  The problem is; how to you charge for what you just did?  That’s where metering and chargeback come into play.  Since the cloud is designed as a “pay-per-use” activity, you need a way to figure out what was used and what your Users should pay.

From the storage perspective, this is easy, right?  Well yes it could be – you could simply total all your storage costs and divide this by all your storage capacity and charged a fixed cost per TB.  Using the example above, if you’ve invested $1,000,000 in a 100TB storage infrastructure, your cost per TB is $10,000 – and you could bill your friend $100,000 for their 10TB of storage.

The problem is that in this model, you are assuming the same requirements for each User, when in reality we all know that there is fast storage, slow storage, big storage, small…well you get the picture.  This kind of level-funding model doesn’t really charge Users for what they are getting.

To try and match costs more exactly, what we usually see IT shops do is divide their storage into 3 or 4 tiers based on loosely defined performance levels, price out each tier, and bill Users according.  This is a step in the right direction, but still far from perfect.  What about automatic storage tiering, for instance.  If an application sees high activity for a few weeks and moves to a higher performance tier, how do account for that?  What if your cloud provides automatic migration of data to reduce latency, how do you account for that?  The answer is automatic metering and a chargeback based on actual usage – something most people haven’t given much thought to, unless of course your business depends on the profits from providing Storage as a Service (STaaS.)

Since the typical IT shop doesn’t need to make a profit, chargeback is used as a means of covering costs, and metering is a way to charging Users fairly .  Metering and chargeback utilities are provided by nearly all major storage providers today – and provide a way to charge Users based on actual usage.  If you are using a chargeback system today (or would like to) and you are moving to a cloud infrastructure, this would be a good time to analyze your methodology.  Here are few questions you should ask:

• Does your chargeback have a sliding scale based on storage tiers?
• How does your  chargeback account for automated tiering?
• Is your chargeback pricing based on logical storage capacity or physical capacity?
• Does your chargeback pricing take high performance IOPS into account?

Metering and chargeback is an area that is sure to see more interest as we move more deeply into cloud and cloud-enabled storage.

For more on this topic, here are  a few storage-brain recommended resources, including User Guides for a couple of popular chargeback utilites:

http://storage-brain.com/wp-content/uploads/papers/IBM_definingframeworkforcloud.pdf#search=

http://storage-brain.com/wp-content/uploads/papers/Storage%20Chargeback%20User%20Guide.pdf

http://storage-brain.com/wp-content/uploads/papers/vCenterChargeback_v10_Users_Guide.pdf

The roots of Flash are in ROM, or Read Only Memory.  In the early days of integrated circuits (this was during the 1970s for our younger readers,) computer microcode was permanently embedded in a set of ROM memory chips.  As the name implies, ROM chips always held the same data, and could not be erased or rewritten.  ROM chipsets worked great – until a code upgrade was needed, in which case the chipset, or the board that held them had to be replaced.

This lead to a breakthrough – EPROM, or Erasable Programmable Read Only Memory.  These chips had a clear window over the silicon and a little sticker over the window, usually denoting the rev of the microcode on the chip.  When it was time to upgrade the code, you pulled the sticker off and put the chip into a little easy-bake oven-looking contraption that had an ultraviolet light in it.  After about an hour under the light, the chip was erased and could be re-coded using a special EPROM programmer.  This was usually done at the factory, but hobbyists (i.e. hackers) could also perform this task with PROM erasers and programmers bought easily through their Computer Shopper magazine and plugged into any PC.

EPROMs allowed you to recycle expensive memory chips, but were still cumbersome to use.  Another breakthrough occurred when the EEPROM became available.  EEPROM stood for Electrically Erasable Programmable Read Only Memory.  The breakthrough here was that the memory chips could be erased by simply putting an electrical charge across the memory banks inside the chip.  In fact, EEPROMS could be erased and reprogrammed without removing them at all.

Along the way, someone figured out that EEPROMs were becoming more than just a way to store microcode, they were becoming solid state data storage devices, like their cousins Random Access Memory (RAM).  Because of the ability to store, erase, and re-store data, EEPROMs are also defined as a form of NV-RAM (Non Volatile Random Access Memory.)

EEPROMs took a huge leap in the 1990s as consumer electronic products started to take hold.  Digital music players, cameras and video recorders all needed substantial storage capacity, but the only viable storage product at the time were 1 inch micro disk drives, which were proving unreliable and expensive.  In response, in 1994, Sandisk announced its Compact Flash (CF) module, based on earlier work by Toshiba and Intel.  CF brought substantial innovations, including a standard packaged module that was a plug-and-play replacement for micro disk-drives and the ability to erase and re-record single memory cells, unlike earlier EEPROMs that needed to be completely erased.

The rest, they say, is history.  Consumer demand took off, and Flash prices plummeted to the point where enterprise storage array vendors took notice.  Could Flash memory modules replace traditional hard drives?  The answer, as you will see, is a definite maybe.

Flash to the Future

For all its greatness, Flash has two major weaknesses.  First of all, like all forms of EEPROM, those little transistors inside the memory module don’t like being flashed.  Eventually, after multitudes of electrical erasures, the transistor gates start to break down and will eventually fail.  Secondly, since you always need to erase, then write, in some cases Flash isn’t very fast.  In fact Flash-based Solid State Drives (SSDs) can be slower than high performance hard disk drives (HDDs) under some conditions.  In other conditions, however, SSD’s are incredibly faster than HDD’s.

What we are seeing today is aggressive innovation in the area of Flash technology that should put it in position to someday altogether replace HDD’s.  The industry is very competitive today with Single Level Cell (SLC) and Multi Level Cell (MLC) Flash designs, as well as NAND and NOR gate wiring in the quest to obtain the right balance of speed, density, reliability, and cost-effectiveness in Flash designs.  Flash software innovations include Wear Leveling and Bad Block Management (BBM) to offset the nasty habit that Flash memory has of wearing out after heavy us.

Are we there yet?  No, not by a long shot.  But surely more innovation is in store.  Stay tuned to these pages as the evolution of Flash continues…

Larry

Memory comes in many flavors. Wikipedia takes us on this walk down memory lane:

While memory is fast, it’s not all that smart. Most random-access memory is simply used as a scratch pad for performing quick compute calculations, or is based on some sort of FIFO logic, meaning “I’ll only keep you in memory until some newer piece of data comes in”

Caching uses memory in a more sophisticated way. A clue to how caching memory works comes from the root of the word – “cacher” a French word meaning “to conceal.” With caching memory, some sort of logic is introduced to decide what data should be held in cache, and which can be ejected. And as the name implies, this is done secretly so that any caching is unknown to the requester of the data.

Caching significantly improves the access time to data when compared to spinning disk or, yes, even SSD.  In a perfect world, all data would be cached but of course costs would be prohibative to allow for this. As an alternative, and because Flash memory is becoming much more affordable, some advances are taking place in the adoption of caching memory.

In an article posted last September, Chris Mellor of The Register talked about six ways to implement flash cache, and also mentioned the vendors who were leading these efforts:

• In server motherboards (Intel)
• In the server PCI bus (Fusion-io, Violin)
•  In the I/O adapter (Adaptec)
•  In the array controller (NetApp)
•  In SSD’s (STEC, Intel)
•  As a replacement for storage arrays (TMS, Sun, Violin)

 Three days after the original article, Mellor posted a 2nd piece, and proclaimed a 7th use for flash cache:

• In the Fibre Channel Fabric (Dataram)

It’s clear to see that the use of Flash cache is spreading quickly.  As I mention in my book:

Enjoy the ride!

Larry

Thanks to microprocessor technology and component miniaturization, storage controllers eventually shrunk down to a single circuit board that plugged into the CPU card cage – first for mainframes, then minicomputers, and finally open systems Unix and Windows servers. As computers got smaller, so did the boards, until the storage controller board was reduced to something you could hold in the palm of your hand. Instead of taking 4 weeks to understand and troubleshoot storage control logic and data flow, it took about 4 minutes to teach someone to replace the storage controller card with a new one. Such was the evolution of computer hardware in those days.

Throughout this period of miniaturization, not much changed as far the role of the storage controller. It simply processed commands sent by the computer system, and stored data on its connected disk drives. Each drive was a separate entity and each controller could manage a small bank of drives, usually up to 16 drives per controller. Until about 1990, this is the way that all disk drives communicated with computers – dumb controllers talking to dumb devices.

Two breakthrough events transformed the dumb storage controller into an intelligent storage array controller. The first was the advent of RAID (Redundant Array of Independent Disks), postulated in the landmark 1988 Berkeley paper. RAID architecture was an amalgamation of many disk drives into a single “logical” drive that processed data quickly and appeared to never fail. This transformation was made possible by the RAID controller – a souped-up storage controller that suddenly had some very useful intelligence. Using RAID, rack-mounted storage arrays with data striped across disk drives could store and retrieve data faster than ever and could continue running even when individual drives within the array stopped working – all done intelligently, without the knowledge or assistance of the computer system.

The second breakthrough occurred with storage virtualization. As far as I can recall, this technology was first made available on the Xiotech Magnitude in the early 2000’s. Before the Magnitude, if you wanted to add disk drives to an array, you had to power the whole thing down, install the drives, and then reconstruct the entire data set, either from another array or from tape backups. In other words, if you had made any plans for the weekend, you might as well cancel them.

Virtual storage made all that work unnecessary by mapping physical drives to logical containers. Want to add capacity? Simply add a couple disk drives to the array (leaving the power on) and map them to the existing drives. Viola! Your new storage capacity magically appeared, and the data was automatically spread out onto the new drives. Suddenly a whole new world of possibilities arose. How about multiple logical partitions within a single array? Yeah, we can do that too!

Today, the storage array controller has grown into a sophisticated, self sustaining entity. The typical array controller manages hundreds of physical disk drives, simultaneously handling RAID configurations, fault recovery and data storage/retrieval functions while optimizing storage capacity with thin provisioning, data deduplication, and snapshot copies. Advanced features such as multi-protocol support, data compression, and self-healing diagnostics are on the verge of becoming mandatory. In their 30+ years of evolution, storage controllers have come a long, long way.

The future looks bright for storage array controllers. Multi-core CPUs provide new processing capability that will allow even more intelligence to be designed throughout. Predictive failure analysis, automated tiering, and intelligent drive spin-down will bring array controllers to new heights. As I mention in my book Evolution of the Storage Brain: “By 2025, it will not be unusual to see a pair of multi-core storage controllers easily managing several petabytes of storage in a single, 19-inch rack.”

If anyone is interested in learning more about the development of storage virtualization techniques, I’d recommend reading Tom Clark’s book “Storage Virtualization: Technologies for Simplifying Data Storage and Management”

http://www.amazon.com/Storage-Virtualization-Technologies-Simplifying-Management/dp/0321262514

Larry

Throughout the 80s, people would write a quick little cron script and backups worked just fine, until the number of servers and the amount of data got a little too big to handle.  Legato was the first independent company to recognize this and offered their Networker backup product as a commercial software application, followed quickly by several dozen competing companies.  These products offered more management than simple scripts to help with backups, things like automatic staggered scheduling, restarting of failed backups, and reports that would tell you if your backup jobs ran OK.

For a while, selling backup software to IT shops was big business.  I suppose it still is today, but instead of dozens of product offerings, the number of backup software vendors has dwindled down to about a half-dozen.  The question I keep asking myself is  how much longer will we need this sort of backup application at all?  Here’s why:

When backup software was king, there were two big things if offered:

1. Tape media management.  Medium-sized IT shops would rotate hundreds of tapes in and out of their data centers.  Large shops would rotate thousands of tapes.  Backup software generally did a nice job of keeping track of which backups were on which tapes, usually with the aid of OCR barcode labels attached to the tape cartridges.
2. File history catalogs.  The backup software would keep a catalog, or database, of all the files it backed up.  Not sure which tape mybigpreso.ppt was backed up to?  You could do a quick search from the backup application and get a list of all the tapes holding your important file, then just click a button to restore it

Backup software hit its peak in the 1990s then started to cool off in the 2000s, but today things have really changed.  First of all, there is much less reliance on tape as a backup medium.  Since 2000, the trend has been towards local disk-based backups and more recently outsourced cloud-based backups.  In either case, the need to track tape media has been reduced or eliminated.  Along with this trend is the growing intelligence of storage arrays.  This new intelligence includes features like the copying of files between storage arrays without server intervention and policy-based data protection.

For over 20 years, backup software applications have been notorious for being finicky and in need of constant attention, believe me I know this from first-hand experience.  They are also known as being very expensive to purchase and maintain.  I believe a new trend is coming soon, and it will revolve around self-backups by intelligent storage arrays. As Jerome Wendt recently blogged, “Server-based backup is dying.”  Although he has a slightly different perspective on the reasons why (based on the fact that virtual servers cannot efficiently back themselves up), he and I are both reaching the same inevitable conclusion.

Larry