When Something Is Simple It Must Be Simple To Prove
Provisioning high-performance storage has always been a chore. Care and concern over spindle count, RAID type, RAID attributes, number of controller arms involved and a long list of other complexities have burdened storage administrators. Some of these troubles were mitigated by the advent of Automatic Storage Management–but not entirely.
Wouldn’t it be nice if the complexity of storage provisioning could be boiled down to but a single factor? Wouldn’t it be nice if that single factor was, simply, capacity? With EMC XtremIO the only factor storage administrators need to bear in mind when provisioning storage is, indeed, capacity.
With EMC XtremIO a storage administrator hears there is a need for, say, one terabyte of storage and that is the entirety of information needed. No more questions about the I/O pattern (e.g., large sequential writes ala redo logging, etc). The Database Administrator simply asks for capacity with a very short sentence and the Storage Administrator clicks 3 buttons in the XtremIO GUI and that’s all there is to it.
Pictures Speak Thousands of Words
I too enjoy the simplicity of XtremIO in my engineering work. Just the other day I ran short on space in a tablespace while testing Oracle Database 12c intra-node parallel query. I was studying a two-node Real Application Clusters setup attached to an EMC XtremIO array via 8 paths of 8GFC Fibre Channel. The task at hand was a single parallel CTAS (Create Table As Select) but the command failed because my ASM disk group ran out of space when Oracle Database tried to extend the BIGFILE tablespace.
Since I had to add some space I thought I’d take a few screen shots to show readers of this blog how simple it is to perform the full cycle of tasks required to add space to an active cluster with ASM in an XtremIO environment.
The following screen shot shows the error I was reacting to:
Since the following example shows host configuration steps please note the Linux distribution (Oracle Linux) and kernel version (UEK) I was using:
The following screenshot shows the XtremIO GUI configuration tab. I selected “Add” and then typed a name and size (1TB) of the volume I wanted to create:
NOTE: Right click the embedded images for greater clarity
The following screenshot shows how I then selected the initiators (think hosts) from the right-hand column that I wanted to see the new volume:
After I clicked “apply” I could see my new volume in my “12C” folder. With the folder construct I can do things like create zero-overhead, immediate, writable snapshots with a single mouse click. As the following screenshot shows, I highlighted “data5” so I could get details about the volume in advance of performing tasks on the host. The properties tab shows me the only information I need to proceed–the NAA Identifier. Once I had the NAA Identifier I moved on to the task of discovering the new volume on the hosts.
Host Discovery
Host discovery consists of three simple steps:
- Multipath discovery
- Updating the udev rules file with a text editor
- Updating udev state with udevadm commands
Multipath Discovery
On both nodes of the cluster I executed the following series of commands. This series of commands generates a lot of terminal output so I won’t show that in this blog post.
# multipath -F ;service multipathd restart ; rescan-scsi-bus.sh -r
After executing the multipath related commands I was able to see the new volume (0002a) on both nodes of the cluster. Notice how the volume has different multipath names (mpathab, mpathai) on the hosts. This is not an issue since the volumes will be controlled by udev:
Updating Udev Rules File and Udev State
After verifying the volumes were visible under DM-MPIO I moved on to the udev actions. The following screenshot shows how I added an ACTION line in the udev rules file and copied it to the other RAC host and then executed the udev update commands on both RAC hosts:
I then could see “/dev/asmdisk6” on both RAC hosts:
Adding The New XtremIO Volume As An ASM Disk
The next task was to use ASMCA (ASM Configuration Assistant) to add the XtremIO volume to the ASM disk group called “DATA”:
As the following screenshot shows the volume is visible as /dev/asmdisk6:
I selected asmdisk6 and the task was complete:
I then saw evidence of ASM rebalancing in the XtremIO GUI Performance tab:
Summary
With EMC XtremIO you provision capacity and that allows you to speak in very short sentences with the application owners that share space in the array.
It doesn’t get any easier than this.
Kevin, as recently as three months ago, Oracle Support recommended that we use /dev/mapper device names for the following reasons:
“The DM-Multipath tool uses three different sets of file names:
Never use /dev/dm-N devices, as they are only intended to be used by the DM-Multipath tool.
Never use /dev/mpath/mpathN devices, because when multipath devices are mounted at boot time , the UDEV subsystem may not create the device nodes soon enough.
Always use /dev/mapper/mpathN devices, as they are persistent and they are automatically created by device-mapper early in the boot process. Therefore these are the device names that should be used to access the multipathed devices. ”
They recommend using ASMLib as well.
Regards,
JT
On Thu, 5 Mar 2015 at 01:07 Kevin Closson’s Blog: Platforms, Databases and
Hi John,
I was with you right up to the mention of ASMLib. ASMLib is a topic of religion at this point. Many see no need for it now that udev provides protection from device name slip and libaio is robust and, besides, if you are aiming to use the 12c ASM Filter Driver then ASMLib is deprecated soon enough (see https://docs.oracle.com/database/121/OSTMG/asminst.htm#OSTMG95729 )
The point of this post was to show a) the fact that capacity is the sole concern when creating an XtremIO volume for use with Oracle Database and b) there are no hoops of fire to jump through at the host level to discover and acquire the storage via ASMCA. I tend to lean on MOS 1528148.1 regarding udev but as the adage goes, “there is more than one way to skin a cat.”
The XtremIO Best Practices guide for Oracle Database use case caters even to those who feel compelled to continue to use ASMLib on modern times. I personally don’t care. ASMLib, PowerPath, DM-MPIO and any combination that includes udev suits me and could suit you as well. Any of these approaches are find with XtremIO. I just have my preferred, simple approach which is what I chose to model this post after.
Kevin,
Happy to take advice from either EMC or Oracle Support.
But reading the manual section you quote, I don’t believe the statement on ASMLib in 12c would even count as “deprecated” let alone “desupported”. (Unless it has been revised very recently.)
https://docs.oracle.com/database/121/OSTMG/asminst.htm#OSTMG95963
I’ve looked up Doc ID 1528148.1. Seems to me it is describing how to use UDEV to map devices, it’s not a recommendation. I don’t mind either way, but I would like robust, consistent advice without any involvement from religion. 🙂
Regards,
JT
John,
Sorry. My point about ASM Filter Driver is that you can’t use with with ASMLib so I applied the word deprecate. I should have said it’s either/or. you’re right.
I still see no purpose for ASMLib but as I said it’s a religious topic and I too, apparently, fall prey to participating 😦
Indeed, I use UDEV for long time as well. I often create one to one mapping between MP disk names and ASM disk names for clearance and better understanding of who ever comes after me. I also create one whole disk partition as per oracle recommendation and to avoid someone running pvcreate or similar command by accident.
Kevin, correct me if I’m wrong by ASMLib would be only mandatory if you want to use ASM with native mode ?
Sve
Sve: “native mode” ?
I believe the reference to “native mode” is in regard to Advanced Format Native Mode (4Kn). On a Linux platform it would indeed be necessary to use ASMLib (or ASMFD) if one were attempting to use ASM with LUNs presented using 4Kn:
http://flashdba.com/4k-sector-size/
And for the record, I am an ASMLib believer 🙂
“And for the record, I am an ASMLib believer ”
…and in spite of that we are still friends…imagine that!
Hi Kevin, would the multipathd restart not affect any Io to the other luns in the system ? Did you shutdown the databsae before presenting the new lun? I missed seeing that step so in case of an always onDB would I need to change the restart to a reload ?
Also how would this change if we are using OVM , I assume the luns for OVM would not get the NAA identifier…