Adding An EMC XtremIO Volume As An ASM Disk With Oracle Database 12c On Linux – It Does Not Get Any Easier Than This.

When Something Is Simple It Must Be Simple To Prove

Provisioning high-performance storage has always been a chore. Care and concern over spindle count, RAID type, RAID attributes, number of controller arms involved and a long list of other complexities have burdened storage administrators. Some of these troubles were mitigated by the advent of Automatic Storage Management–but not entirely.

Wouldn’t it be nice if the complexity of storage provisioning could be boiled down to but a single factor? Wouldn’t it be nice if that single factor was, simply, capacity? With EMC XtremIO the only factor storage administrators need to bear in mind when provisioning storage is, indeed, capacity.

With EMC XtremIO a storage administrator hears there is a need for, say, one terabyte of storage and that is the entirety of information needed. No more questions about the I/O pattern (e.g., large sequential writes ala redo logging, etc). The Database Administrator simply asks for capacity with a very short sentence and the Storage Administrator clicks 3 buttons in the XtremIO GUI and that’s all there is to it.

Pictures Speak Thousands of Words

I too enjoy the simplicity of XtremIO in my engineering work. Just the other day I ran short on space in a tablespace while testing Oracle Database 12c intra-node parallel query. I was studying a two-node Real Application Clusters setup attached to an EMC XtremIO array via 8 paths of 8GFC Fibre Channel. The task at hand was a single parallel CTAS (Create Table As Select) but the command failed because my ASM disk group ran out of space when Oracle Database tried to extend the BIGFILE tablespace.

Since I had to add some space I thought I’d take a few screen shots to show readers of this blog how simple it is to perform the full cycle of tasks required to add space to an active cluster with ASM in an XtremIO environment.

The following screen shot shows the error I was reacting to:

2015-02-23-pic1

Since the following example shows host configuration steps please note the Linux distribution (Oracle Linux) and kernel version (UEK) I was using:

linux-release

The following screenshot shows the XtremIO GUI configuration tab. I selected “Add” and then typed a name and size (1TB) of the volume I wanted to create:

NOTE: Right click the embedded images for greater clarity

2015-02-23-pic2

The following screenshot shows how I then selected the initiators (think hosts) from the right-hand column that I wanted to see the new volume:

2015-02-23-pic4

After I clicked “apply” I could see my new volume in my “12C” folder. With the folder construct I can do things like create zero-overhead, immediate, writable snapshots with a single mouse click. As the following screenshot shows, I highlighted “data5” so I could get details about the volume in advance of performing tasks on the host. The properties tab shows me the only information I need to proceed–the NAA Identifier. Once I had the NAA Identifier I moved on to the task of discovering the new volume on the hosts.

 

2015-02-23-pic5

Host Discovery

Host discovery consists of three simple steps:

  1. Multipath discovery
  2. Updating the udev rules file with a text editor
  3. Updating udev state with udevadm commands

Multipath Discovery

On both nodes of the cluster I executed the following series of commands. This series of commands generates a lot of terminal output so I won’t show that in this blog post.

# multipath -F ;service multipathd restart ; rescan-scsi-bus.sh -r

After executing the multipath related commands I was able to see the new volume (0002a) on both nodes of the cluster. Notice how the volume has different multipath names (mpathab, mpathai) on the hosts. This is not an issue since the volumes will be controlled by udev:

2015-02-23-pic7

Updating Udev Rules File and Udev State

After verifying the volumes were visible under DM-MPIO I moved on to the udev actions. The following screenshot shows how I added an ACTION line in the udev rules file and copied it to the other RAC host and then executed the udev update commands on both RAC hosts:

2015-02-23-pic9

I then could see “/dev/asmdisk6” on both RAC hosts:

2015-02-23-pic10

Adding The New XtremIO Volume As An ASM Disk

The next task was to use ASMCA (ASM Configuration Assistant) to add the XtremIO volume to the ASM disk group called “DATA”:

2015-02-23-pic8

As the following screenshot shows the volume is visible as /dev/asmdisk6:

2015-02-23-pic11

I selected asmdisk6 and the task was complete:

2015-02-23-pic12

I then saw evidence of ASM rebalancing in the XtremIO GUI Performance tab:

2015-02-23-pic15

 

Summary

With EMC XtremIO you provision capacity and that allows you to speak in very short sentences with the application owners that share space in the array.

It doesn’t get any easier than this.

9 Responses to “Adding An EMC XtremIO Volume As An ASM Disk With Oracle Database 12c On Linux – It Does Not Get Any Easier Than This.”


  1. 1 John Thomas March 5, 2015 at 9:45 am

    Kevin, as recently as three months ago, Oracle Support recommended that we use /dev/mapper device names for the following reasons:

    “The DM-Multipath tool uses three different sets of file names:

    Never use /dev/dm-N devices, as they are only intended to be used by the DM-Multipath tool.

    Never use /dev/mpath/mpathN devices, because when multipath devices are mounted at boot time , the UDEV subsystem may not create the device nodes soon enough.

    Always use /dev/mapper/mpathN devices, as they are persistent and they are automatically created by device-mapper early in the boot process. Therefore these are the device names that should be used to access the multipathed devices. ”

    They recommend using ASMLib as well.

    Regards,

    JT

    On Thu, 5 Mar 2015 at 01:07 Kevin Closson’s Blog: Platforms, Databases and

    • 2 kevinclosson March 5, 2015 at 1:04 pm

      Hi John,

      I was with you right up to the mention of ASMLib. ASMLib is a topic of religion at this point. Many see no need for it now that udev provides protection from device name slip and libaio is robust and, besides, if you are aiming to use the 12c ASM Filter Driver then ASMLib is deprecated soon enough (see https://docs.oracle.com/database/121/OSTMG/asminst.htm#OSTMG95729 )

      The point of this post was to show a) the fact that capacity is the sole concern when creating an XtremIO volume for use with Oracle Database and b) there are no hoops of fire to jump through at the host level to discover and acquire the storage via ASMCA. I tend to lean on MOS 1528148.1 regarding udev but as the adage goes, “there is more than one way to skin a cat.”

      The XtremIO Best Practices guide for Oracle Database use case caters even to those who feel compelled to continue to use ASMLib on modern times. I personally don’t care. ASMLib, PowerPath, DM-MPIO and any combination that includes udev suits me and could suit you as well. Any of these approaches are find with XtremIO. I just have my preferred, simple approach which is what I chose to model this post after.

      • 3 John Thomas (@jt2354) March 6, 2015 at 2:52 am

        Kevin,

        Happy to take advice from either EMC or Oracle Support.

        But reading the manual section you quote, I don’t believe the statement on ASMLib in 12c would even count as “deprecated” let alone “desupported”. (Unless it has been revised very recently.)

        https://docs.oracle.com/database/121/OSTMG/asminst.htm#OSTMG95963

        I’ve looked up Doc ID 1528148.1. Seems to me it is describing how to use UDEV to map devices, it’s not a recommendation. I don’t mind either way, but I would like robust, consistent advice without any involvement from religion. 🙂

        Regards,

        JT

        • 4 kevinclosson March 6, 2015 at 12:48 pm

          John,

          Sorry. My point about ASM Filter Driver is that you can’t use with with ASMLib so I applied the word deprecate. I should have said it’s either/or. you’re right.

          I still see no purpose for ASMLib but as I said it’s a religious topic and I too, apparently, fall prey to participating 😦

  2. 5 sgyurov March 6, 2015 at 3:25 am

    Indeed, I use UDEV for long time as well. I often create one to one mapping between MP disk names and ASM disk names for clearance and better understanding of who ever comes after me. I also create one whole disk partition as per oracle recommendation and to avoid someone running pvcreate or similar command by accident.

    Kevin, correct me if I’m wrong by ASMLib would be only mandatory if you want to use ASM with native mode ?

    Sve

  3. 9 robinsc August 9, 2016 at 11:55 pm

    Hi Kevin, would the multipathd restart not affect any Io to the other luns in the system ? Did you shutdown the databsae before presenting the new lun? I missed seeing that step so in case of an always onDB would I need to change the restart to a reload ?
    Also how would this change if we are using OVM , I assume the luns for OVM would not get the NAA identifier…


Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s




DISCLAIMER

I work for Amazon Web Services. The opinions I share in this blog are my own. I'm *not* communicating as a spokesperson for Amazon. In other words, I work at Amazon, but this is my own opinion.

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 2,961 other followers

Oracle ACE Program Status

Click It

website metrics

Fond Memories

Copyright

All content is © Kevin Closson and "Kevin Closson's Blog: Platforms, Databases, and Storage", 2006-2015. Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Kevin Closson and Kevin Closson's Blog: Platforms, Databases, and Storage with appropriate and specific direction to the original content.

%d bloggers like this: