Its time to give credit where credit is due, one thing all of IT professionals have in common is when we find a product we love to use and manage we like to share that product with everyone we know, this leads me to NimbleStorage.
I ran into Nimble Storage a couple of years ago. A couple of guys came into my office (Jay and Eddie) and started to talk about how they have a product that can replace shelves of disk with only a 3U unit full of High Capacity SATA Drives, and 4 SSD Drives just used as a read Cache, plus give me WAN-Efficient Replication, 90 Days of Backup on Disk, Space efficient Snapshots, Compression (that increases performance) and High IOPs with low latency….. Ok thinking to myself these guys must have drank the Kool-Aid. Everything they say it can accomplish I WANT, can it really be true?
A year or so went by and I ended up changing Roles and moved to a new company, they needed a new SAN. What did I want in a SAN, everything that Jay and Eddie spoke of, I wanted that….. was it the Kool-Aid or was it real, I was going to find out before I made the choice to go with another Vendor. Did my research online, and couldn’t find allot, but anything I did find was good, only comment in the negative was they are new, and beware of that. All companies got to start somewhere, Apple, Microsoft, Dell, and Veeam just to name a few, would I be willing to put my name to their vision and product?
Called up NimbleStorage and brought them in for a chat or two. Not only are they still promising the same thing, but now since a year has went past they are growing bigger, and now on the road map they have scalability in mind! I go ahead with a POC, within a couple of days I have a Nimble CS220 sitting in my Datacenter, a day or two later I get Eddie onsite, and we have the Nimble Array up and running in about 30 Minutes and ready to add some VM’s to it! Now is the time I get to see if all the hype is true. For the testing I spun up a VM, then we ran a SQLIO from the Server, we let the test run, when we were all said and done, we seen MAX IOPS at just over 18K!!! This was also while we were updating the Array to the newest software version. Lets just say I went with NimbleStorage and not regretting it one little bit, at the moment I have all my Servers moved to the Nimble Array, a Total of 60 servers running a combination of different workloads, from Exchange, SQL, File, Web, and More….. The total capacity of these servers were about 8 TB’s yet I am only using 4.2 TB’s, almost a 50% increase in capacity, while also keeping 60 Days of 1 Hour Snapshots on the Array, reducing my RPO, and RTO.
On to the Meat and potato’s
My Testing
Below I will just include some of the screen shots of the testing I did with my Nimble CS220. This is not all the testing I have done, but just some that are worth sharing. Don’t really want to cover the blog posting with SQLIO data.
The below Iometer test were running at the same time across two different Host. (Only had 1Gbps iSCSI connected at the time)
Reporting from the Console of the Nimble during the Test
One that was worth sharing.
![clip_image002[6]](https://rickrbyrne.files.wordpress.com/2012/10/clip_image0026.jpg "clip_image002[6]")
2 1Gbps iSCSI connections using VMware iSCSI software initiator.
![clip_image002[8]](https://rickrbyrne.files.wordpress.com/2012/10/clip_image0028.jpg "clip_image002[8]")
Recommendations for Performance
Most of these are the same for most iSCSI based SAN’s but figured it was something nice to note.
Multi Switch Connectivity
Network
VMware
NimbleStorage Product Information
Performance
Now you need to ask yourself a question, how can Nimble get such high IOPS from 8 SATA Drives (We all know it’s not all about the IOPS Nimble also offers, more throughput and lower latency). The technology that makes that possible is the CASL Architecture (Cache Accelerated Sequential Layout). CASL was designed from the ground up to leverage the best of flash and disk, it eliminates compromise between performance, capacity, and data management.
Universal Compression: Variable-size Blocks enable fast inline compression. Elimination of read-modify-write penalty allows compression of all applications.
smart Caching: converting random writes to sequential writes minimizes “write amplification” allowing use of MLC SSDs. No RAID overhead. Data on flash is compressed and metadata in cache accelerates all reads
Write Optimized Layout: Random writes always organized into large sequential stripes. Use of low cost high-density HDDs coupled with compression lowers cost substantially
The way the process works in my own words (Please don’t fully quote this 
)
1. VM Disk writes data to a 4.5 MB block directly in NVRAM. This data is compressed and can be variable size. The 4.5 MB block now holds 1000-3000 IO’s per block.
2. Once the 4.5 MB block is full, it gets written to the slower High density HDD’s in a sequential striped pattern.
3. VM disk reads data, if it is held in the Cache (From recent read or write) it will get sent to the VM, and if more data is needed it will be pulled from the Disk’s below and get pre-cached. Cache hit rate is usually 75%, it basically comes down to how you manage your data. You don’t want Transaction Logs, Log files, and things of that nature to remain in cache. You can control this with protection profiles. If you plan your LUNS correctly you can easily get a better cache rate than I have mentioned above. The nice thing about the Nimble is it comes with a large cache.
Features
Dynamic Caching
Reads active data from flash cache, which is populated on writes or first read. Accelerate read operations, with sub-millisecond latency
Write-Optimized Data Layout
Coalesces random writes and sequentially writes them to disk as a full stripe. Accelerate writes as much as 100x, and get sub-millisecond latency and optimal disk utilization
Universal Compression
Always-on inline compression for all workloads. Reduce capacity needs 30-75% depending on the workload with no performance impact
Thin Provisioning
Allocates disk space to a volume only as data is written. Pool storage, share free space and maximize utilization
Scale-Out Clustering
Combine multiple arrays into a scale-out storage cluster. Linearly scale beyond performance and capacity. limits of a single storage array while managing these multiple arrays as one
Instant Snapshot and Recovery
Backup and restore data using point-in-time, space-efficient snapshots taken at regular intervals. Retain months of frequent snapshots (improving RPO) with no performance impact–eliminating backup windows and speeding up restores (improving RTO)
Efficient Replication
Copies compressed, changed data to the secondary site for disaster recovery. Deploy affordable and verifiable disaster recovery and efficiently backup remote sites over the WAN
Zero-Copy Clones
Create copies of existing active volumes without needing to copy data. Create clones in seconds and save disk space – ideal for VDI and test/development
Custom Application Profiles
Pre-defined policies for block size, caching, compression and data protection for Microsoft applications and VMware. Eliminate the need for manual tuning of storage including data protection configuration
Windows VSS Enablement
Nimble Storage Driver for the Microsoft VSS framework for consistent backup. Take application-consistent backups and simplify data protection for Exchange and SQL Server
VMware Integration
Monitor, provision and take snapshots from VMware vCenter .Manage storage from vCenter and take consistent backups of virtual machines
VMware Site Recovery Manager Adapter
Support disaster recovery automation for VMware including failover/failback. Simplifies disaster recovery, including testing failover/failback
Products
Below is a table listing all the configurations you can get from NimbleStorage
The difference between the CS2XX and the CS4XX is the CS4XX has upgrading computing on the controllers to allow for even more IOPS and performance compared to CS2XX
Scale to Fit
We all need a product that can Scale, Nimble Storage now offers that. You can scale capacity by adding additional storage shelves, all arrays but the CS210 support up to 3, CS210 is 1. Scale up performance by upgrading compute (Add a CPU) to get greater throughput and IOPS, and expanding cache with larger SSDs to handle more active data. You can also scale out both performance and capacity by non-disruptively combining multiple arrays into one easy to manage scale-out cluster.
Scale-Out Cluster
With this you are getting dynamic storage pools that strip data across the arrays and is automatically rebalanced as you expand or shrink pools, this is all done behind the scenes and makes sure you don’t have one array full while one is empty. Management is done under a single pane of glass, meaning you get to manage the entire cluster from one central console. Host will get parallel access to the array which will get the best performance possible and reduce latency.
No matter which solution you decide to go with, you can use your existing array you have in-place, and also be able to upgrade any components without any downtime.
Some vendors have release Cluster’s for their arrays, but if you are a current customer it requires you to fork lift all your investment to benefit from the new feature set.
Support
I have had no issues with Support, so far it has been amazing. Now.. I haven’t had any major issues, but anything that did come up they were alerting me before I had the time to even look. Anytime I have called for general questions and inquiries I have had a person on the phone within minutes with no phone transfer games.
To add to their support they have proactive wellness, these tools automatically resolved 75 percent of the issues without you having to do a thing. Basically the array calls home, and reports any configuration issues or abnormal operating conditions. You also have the ability to enable SSH tunnels, this way support can get direct access to your array and do their magic.
Conclusion
Was this a love letter to Nimble? No I just wanted to give some credit, I love the product. I did doubt them in the beginning, should I have doubted them, yes. There are so many start-up companies you have to do your research and make sure you are making the right choice for your company, but in the end you need to take a chance, and I did. Do I regret going with NimbleStorage, not one little bit, I am the system administrator who has to wear many hats, with Nimble I wear the storage hat, but I don’t have to put it on very much because it just works. If you are in the Market for new Storage, please take a moment to look at Nimble, they can even be a nice add on to your existing storage, especially VDI based projects. Maybe I drank the Kool-Aid, it taste so good, and I just want more and have a shelf on order to start scaling.
In the next few weeks I will put up a post or two related to the operation of the NimbleStorage Array. This post has went on long enough!
If you made it through the full post, thanks for reading.
The Cloud Eh!? 
Curious to know how it handles a non-cache friendly workload, thats not a lot of spindles to use when the SSD’s are not in play, plus its SATA. That has always been my curious observation when it came to the Nimble and other hybrid type arrays. Also, I’d like to see some disparate workloads which are very easy to replicate using VMware IO Analyzer. Say run a sample Exchange, SQL, and File server concurrently instead of single host based load generation. I’m not too keen on SQLIO, IIRC the data file it uses is all commas.
Nice writeup all the same.
Thanks for reading.
I totally agree with your concern and it was one I shared. Nimble was concerned with the same factor when they were thinking of CASL. CASL has the ability to provide Universal Application compression, what this means is that when you are writing all your data get’s put into a 4.5 MB block, all variable sized data will end up in this block, and it will stay opened until it is full and then get’s written to disk. Now this 4.5 MB block is now filled with 1000 – 3000 IO’s, on read, if this data is still in the cache it will be pulled from there, and you will get the benefit of the SSD speed. However if the data is not in the Cache, it will pull it from disk, but it is one big pull and now it resides in the cache, and you are not just bringing up 1 IO your are bringing up 1000-3000 IO’s. Also the Cache is just configured in a JBOD, meaning you are not losing a disk for RAID protection, and you are also not having to be concerned with a Disk failing and bringing down your cache, it will only decrease the size until you replaced the failed drive. The cache is also running on MLC’s not SLC’s which even saves you more money. They also have the ability to do predictive cache which in most times will get the data you need. In my production SAN I am seeing a 75% cache hit, which is great. Plus I don’t have any policies in place to stop the Log’s and things of that nature from caching. I am planning that change when I get my shelf to add more capacity and give me a little more room for better LUN design and VM placement.
One last thing, allot of Vendors out there have tried to use SSD’s the same way they used Mechanical Spindles, this is why they require SLC’s because of the abuse the SSD’s take while using them. Nimble on the other hand had the ability to build their array around flash instead of trying to build SSD into an File System that was tailored for mechanical spindles.
Have a good one.
Rick – We really appreciate the positive feedback,
Nice write up. Nimble appears to be a storage sector disruptor. I’m jumping in headfirst on Monday with a CS 220. Hope I don’t regret it.
I didn’t regret it one bit!
Everything you said was true. It just works and it works well. The hype is as real as the performance. You can be up and running on this puppy in a day if you are sharp on the networking side. We spent WAY more time on networking and VMware setup than we did on the SAN setup. The Nimble can be online and ready to serve up storage in about 30 minutes.