NVMe, PCIe and SSD Technology Review

pcie is coming fastest storage devices

Although NVMe technology has been around since 2010, the meaningful use of this technology is only now becoming widely accepted in the technology industry.  In the storage world, NVMe is a game changer unlike anything we have seen in quite some time.

Henry Ford once said that if he had asked people what they wanted, they would have asked for “faster horses.”

This implies that the founders of NVMe didn’t want to just improve the current storage process and performance, they set out to disrupt the storage market.  They knew that the current storage systems had their limits and that it didn’t make sense to provide a short-term solution to this problem, a longer term, scalable solution was necessary.

Why did we need something better?

Data is growing, there is no doubt about that.  With the advent of higher resolution video and photos, the “Internet of Things,” social media and “compliance and security,” we are beginning to speak in Exabytes and Zettabytes! Both consumers and enterprises are generating more and more data each year and expect faster response times when accessing or analyzing that data.

hard drive rpm explained

Over time we have seen that most of a computer’s resources have increased their performance envelop significantly, this includes CPU, Memory and Networking.  What has not increased performance however is spinning disk drives. The SATA based Hard Drives for example can only spin at 5400 or 7200 RPM’s and this has been consistent for the last 15 years. Along the way, the introduction of 10,000 and 15,000 RPM drives have certainly increased capacities, decreased seek time, and reduced the latency.  Combine this with the concept of RAID or striping and mirroring and you were able to create a moderately faster disk storage subsystem.

However, the point of diminishing returns is upon us and physics will simply not allow the drives to spin faster without overheating and crashing.  The disk subsystem had become the infamous bottleneck in the computer.

The answer – Solid Stat Drives (SSD).  An SSD drive is simply is a hard drive, but data is stored on interconnected flash memory chips that retain the data even when no power is present.  SSD drives can support several times the IOPS of its HDD predecessor

The original SSD’s were essentially retrofitted into the disk storage system bus by maintaining the original adapters that were used for SATA hard drives. This was great for compatibility, but it was a huge prohibitor of the potential speeds and response time of the SSD’s. The original SSD’s were attached to the SATA bus adapter using AHCI interface, circa 2004! They began with bus speeds of 1.5Gb/sec, 3.0Gb/sec and then 6.0Gb/sec, but it still wasn’t fast enough to match the speed of the SSD drives. (http://forums.crucial.com/t5/Crucial-SSDs/Why-do-i-need-AHCI-with-a-SSD-Drive-Guide-Here-Crucial-AHCI-vs/td-p/57078)

The need to fully exploit the performance capabilities of an SSD was really becoming necessary in the latter part of the 2000’s and the idea of a new bus interface was born with the advent of NVMe. NVMe stands for Non-Volatile Memory Express and runs over the PCIe bus increasing the performance of the SSD ten-fold. NVMe was developed by several large storage manufacturers helping to endorse the adoption. Companies such as Intel, Samsung and LSI participated to develop the standards and broaden the acceptance in the industry.

Capacities

Now that NVMe is becoming more mainstream, the manufacturers are broadening as they quench the thirst for the demand of speed. There are now more than a dozen drives available today, which provides the consumer with not only options but competition and thus a more affordable price. Samsung and Intel both make a consumer based “M.2” (Gumstick) 512GB drive. Samsung also manufacturers a 2TB drive and there are others such as Corsair, Plextor, Kingston and Patriot.

As for the enterprise landscape, the NVMe PCIe drives (half height/half length) are available in 3.82TB capacities and provide over 115K IOPS and 3000MB/sec throughput.

What About Price?

So, do you just run out and purchase NVMe? You will certainly want to ensure that your BIOS in the target system is capable of supporting the NVMe technology that you choose.  Whether you are purchasing the drives for personal/consumer use or for the enterprise, you will find that there are very different price floors across the choices you have. The 2016 saw plenty of new contenders in the NVMe realm. Many of these still fail to match the performance of the Intel SSD 750 and Samsung 950 Pro in consumer testing and their prices began to come down and then it started to head back up due to the NAND shortage. More drives are in the works, and time will tell if any of those can displace Samsung and Intel in the future.

The NAND chips make up a substantial portion of the total SSD price, and fundamentally there usually isn’t much of a difference between a SATA drive with 512GB V-NAND and an M.2 NVMe drive with the same 512GB V-NAND. NVMe requires a new controller and firmware, and there are associated R&D costs, but long-term NVMe prices should get a lot closer to SATA SSD prices, with a moderate premium. Economies of scale are still a factor, however, until more systems use NVMe SSDs, prices will stay higher.

Typically, widespread adoption will force a price drop over time, but NAND technology which is used in SSD’s is in high demand amongst mobile phones, laptops and PCs as well as within the enterprise datacenter. In addition to the increasing demand of SSD everywhere, there is another factor, 3D NAND.

What is 3D NAND?

Much like NAND, it is a flash memory technology in which the memory cells are stacked vertically, creating multiple layers.  The idea is to be able to increase density at a lower price point.  While manufacturers turn to 3D NAND, allowing them to scale beyond the capacities of the 2D/Planar NAND, they are finding that reliability issues as well as early limitations on the sizes of the drives.

3D NAND is forcing manufacturers to convert their factories from NAND which requires retooling, thus forcing them to take the factories offline during the conversion, limiting the supply for a period of time.  As the manufacturers continue to improve yields, increase capacity and decrease the cost, we should see a drop overall in 3D NAND technology.

Which types of applications are best suited for NVMe PCIe technology?

There are actually several consumer and enterprise applications that will benefit, in fact there are very few that would not benefit. Any time that the data being accessed (Read or Written) can be brought closer to the CPU and delivered faster across the system bus, your application performance will improve. In this case, if your application requires disk access, you will be able to retrieve and write the data at a much higher rate. In the enterprise space, ERP, Email, High Performance Computing, general OLTP and OLAP type of applications will benefit as there is plenty of IO that occurs here. In the consumer world, gaming consoles, gaming PC’s and mobile phones will all benefit as the physical capacity decreases while at the same time providing significant performance boosts.

Will NVMe replace SAS and SATA based disk drives?

Very likely someday, the NVMe (or variation) drive will replace the older technology, but there are thousands of systems out there running just fine with SATA and SAS. If those applications do not benefit from the SSD technology then why spend more money and gain no additional benefit?

As the cost of NVMe decreases, the value of the SATA and SAS technology certainly will wane, but it may be very costly to replace systems that are running applications that organizations still depend on every day. The three technologies, NVMe, SATA and SAS will continue to co-exist, possibly even in the same datacenter or even the same server. A visual representation comparing the three technologies is below and as you can see, there are significant performance differences especially in the number of queues and commands per queue that can be handled.

The storage IO capabilities of flash can now be streamed across PCIe buses faster, to enable modern processors to complete more useful work in less time, resulting in greater application productivity.

NVMe has been designed from the ground up with more and deeper queues, supporting a larger number of commands in those queues. This in turn enables the SSD to better utilize command execution for much higher concurrent IOPS. NVMe will co-exist along with SAS, SATA and other server storage IO technologies for some time to come. But NVMe will be at the top tier of storage as it takes full advantage of the speed and low latency of flash while complementing the potential of multi-core processors that can support the latest applications.

nvme pcie

Like the robust PCIe physical server storage IO interface it leverages, NVMe provides both flexibility and compatibility. It removes complexity, overhead and latency while allowing far more concurrent I/O work to be accomplished. Those on the cutting edge will embrace NVMe rapidly. Others may prefer a phased approach.

Some environments will initially focus on NVMe for local server storage IO performance and capacity available today. Other environments will phase in emerging external NVMe flash-based shared storage systems over time.

Planning is an essential ingredient for any enterprise. Because NVMe spans servers, storage, IO hardware and software, those intending to adopt NVMe need to take into account all ramifications. Decisions made today will have a big impact on future data and information infrastructures.

Key questions should be, how much speed does your application need, and how do growth plans affect those requirements? How and where can you maximize your financial return on investment (ROI) when deploying NVMe and how will that success be measured?

Mackenzie Hewitt, 

LEAVE A REPLY

Please enter your comment!
Please enter your name here