December 5, 2006 9:43 pm
Using strace(1) to Study Database Writer on Linux
This is a short blog entry for folks that are interested in Oracle10g’s usage of libaio asynchronous I/O routines (e.g., io_submit(2)/io_getevents(2)). For this test, I set up Oracle10g release 10.2.0.1 on Red Hat 4 x86_64. I am using the cluster filesystem bundled with the PolyServe’s Database Utility for Oracle, but for all intents and purposes I could have used ext3.
The workload is a simple loop of INSERT INTO SELECT * FROM statements to rapidly grow some tables thereby stimulating Database Writer (DBWR) to flush modified SGA buffers to disk. Once I got the workload running, I simply executed the strace command as follows where <DBWR_PID> was replaced with the real PID of the DBWR process:
$ strace -o dbw -p <DBWR_PID>
NOTE: Using strace(1) imposes a severe penalty on the process being traced. I do not recommend using strace(1) on a production instance unless you have other really big problems the strace(1) output would help you get under control.
The second argument to the io_submit(2) call is a long integer that represents the number of I/O requests spelled out in the current call. The return value to an io_submit(2) call is the number of iocb’s processed. One clever thing to do is combine grep(1) and awk(1) to see what degree of concurrent I/O DBWR is requesting on each call. The following screen shot shows an example of using awk(1) to select the io_submit(2) calls DBWR has made to request more than a single I/O. All told, this sould be the majority of DBWR requests.
NOTE: You may have to right click->view the image. Some readers of this blog have reported this. Sorry
Another way to do this is to anchor on the return to the io_submit(2) call. The following screen shot shows an example of grep(1) to selecting only the io_submit(2) calls that requested more than 100 I/O transfers in a single call.
What File Descriptors?
When io_submit(2) is called for more than one I/O request, the strace(1) output will string out details of each individual iocb. Each individual request in a call to io_submit(2) can be for a write to a different file descriptor. In the following text grep(1)ed out of the strace(1) output file, we see that Oracle requested 136 I/Os in a single call and the first 2 iocbs were requests to write on file descriptor 18.:
io_submit(182926135296, 136, {{0x2a973cea40, 0, 1, 0, 18}, {0x2a973e6e10, 0, 1, 0, 18}
What About ASM?
You can run ASM with libaio. If you do, you can do this sort of monitoring, but you wont really be able to figure out what DBWR is writing to because the file descriptors will just point to raw disk. ASM is raw disk.
Posted by kevinclosson
Categories: DBWR Performance, oracle, Oracle Asynchronous I/O, Oracle DBWR
Tags:
You must be logged in to post a comment.
Mobile Site | Full Site
Get a free blog at WordPress.com Theme: WordPress Mobile Edition by Alex King.
[…] Linux, asynchronous I/O requests are submitted through the io_submit() call. (Kevin Closson once wrote about tracing this call.) This function essentially puts the I/O request into a queue which is managed by the Linux kernel. […]
By Oracle I/O and Operating System Caching : Ardent Performance Computing on February 1, 2008 at 5:05 am
[…] This is not a post about why someone would want to deploy Oracle with a mix of files with varying support for asynchronous I/O. It is just a peek at how Oracle10g handles it. This blog post is a continuation of yesterday’s topic about analyzing DBWR I/O activity with strace(1). […]
By Analyzing Asynchronous I/O Support with Oracle10g « Kevin Closson’s Oracle Blog: Platform, Storage & Clustering Topics Related to Oracle Databases on May 27, 2010 at 3:31 am