Kevin Closson's Blog: Platforms, Databases and Storage

Guy Harrison has been blogging his findings regarding solid state disk testing/performance with Oracle. Guy’s tests and reports are very thorough. This is a good body of work. The links to follow are at the bottom of this post.

Before reading, however, please consider the following thoughts about solid state disk as pertaining to Oracle I/O performance and flash:

1. Oracle Database Smart Flash Cache (DSFC) is flash storage with libC/libaio physical I/O to “augment” the SGA. When using DSFC you will sustain DBWR writes even if your application only uses the SELECT statement. Few people are aware of this fact. The real SGA buffer pool becomes “L1” cache and DBWR spills clean blocks to the L2 (DSFC) where subsequent logical I/O (cache buffers chain hits) will actually require a physical read from flash. Sessions cannot access buffered data in the DSFC. Blocks have to first be read from flash into the SGA before the session can get along with its business. I have never seen DSFC work effectively. I have, on the other hand, seen a whitepaper showing read-intensive “oltp” serviced by an SGA that is much smaller than available RAM on the system but and augmented by flash. However, the paper is really just showing that flash reads are better than reads from an under-configured hard disk farm. I’ll blog about that paper soon.
2. Database Smart Flash Cache, Really?  Don’t augment DRAM until you’ve maxed out DRAM.  Do I honestly aim to assert that augmenting nanosecond operations with millisecond operations it non-optimal?, that is my assertion.
3. DBWR spills to DSFC aren’t charged to sessions, but such activity does take system bandwidth.
4. If you find a real-life workload that benefits from DSFC please let me know.
5. Redo Log physical I/O is a well-suited for round, brown spinning disks. Just don’t melt the disks with DBWR random writes and expect the same disks to favor the occasional large sequential writes issued by LGWR. Watch your log file parallel write (LFPW) wait events as a component of log file sync (LFS) and you’ll usually find that LFS is a CPU problem, not a LFPW problem. Read this popular post for more on that matter.
6. Don’t expect much performance increase, in an Exadata environment, from the new “Exadata Smart Flash Log” (ESFL) feature. It may indeed smooth out LFPW service times, and that is a good thing, but the main benefit Smart Flash Log delivers to Exadata customers is relief from the HDD controller starvation (not to mention the imbalance of processors to spindles) that happens in Exadata when DBWR and LGWR are simultaneously demanding IOPS from the limited number of spindles in Exadata standard configurations. Remember, Exadata’s design center is read bandwidth, not write IOPS. A full rack X2 Exadata configuration can only sustain on the order of 25,000 mirrored random writes per second. The closer one pushes Exadata to that 25K IOPS figure the more trouble LGWR has with log flushes. That is, Exadata Smart Flash Log (ESFL) is really just a way to flow LGWR traffic through different adaptors (PCI Flash). In fact, simply plugging in another LSI HDD controller with a few SATA drives dedicated to redo streaming writes would actually do quite well if not as well as ESFL. That is a topic for a different post.
7. EMC Fully Automated Storage Tiering (FAST) does not muddle around with redo because redo does really well with spinning disks. Hard disk drives do just fine with large sequential writes.
8. Oracle Database 11g Release 2 added the ability to specify the block size for a redo log. If you do feel compelled to flush redo to solid state I recommend you crack open the documentation for the BLOCKSIZE syntax and add some 4K blocking-factor logs. I made this point on Guy’s blog comment section. I’m not sure if his tests were 4K block size or if they were flushing redo with 512-byte alignment (which flash really doesn’t favor).

And now references to Guy Harrison’s posts:

http://guyharrison.squarespace.com/blog/2011/10/27/using-flash-disk-for-redo-on-exadata.html

http://guyharrison.squarespace.com/ssdguide/04-evaluating-the-options-for-exploiting-ssd.html