Fault Injection Testing. Spurious Space Depletion? Sure, Why Not?

When file systems run out of space bad things happen. We like to investigate what those “bad things” are but to do so we have to create artificially small installation directories and run CPU-intensive programs to deplete the remaining space. There is a better way on modern Linux systems.

If you should find yourself performing Linux platform fault-injection testing you might care to add spurious space free failures. The fallocate() routine immediately allocates the specified amount of file system space to an open file.  It might be interesting to inject random space depletion in such areas as Oracle Clusterware (Grid Infrastructure) installation directories or application logging directories. Could a node ejection occur if all file system space immediately disappeared? What would that look like on the survivors? What happens if large swaths of space disappear and reappear? Be creative with your destructive tendencies and find out!

 


#include <asm/unistd.h>
#include <errno.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/stat.h>
#include <sys/syscall.h>
#include <sys/types.h>
#include <unistd.h>


int main(int argc, char *argv[])
{
long int sz;
char *fname;
int ret,fd;

if (argc != 3)
{
fprintf(stderr, "usage: %s file new-size-in-gigabytes\n", argv[0]);
return(-1);
}

fname = argv[1];
sz   = atol(argv[2]);

if ((ret = (fd = open(fname, O_RDWR | O_CREAT | O_EXCL, 0666)))  == -1 ) {
perror("open");
return(ret);

}
if ( (ret = fallocate( fd, 0, (loff_t)0, (loff_t)sz * 1024 * 1024 * 1024 )) != 0 ){
perror ("fallocate");
unlink( fname );
}

close(fd);
return ret;
}

 

 

 

#
# cc fast_alloc.c
#
# ./a.out
usage: ./a.out file new-size-in-gigabytes
#
# df -h .
Filesystem Size Used Avail Use% Mounted on
/dev/sdc 2.7T 1.6T 1.2T 57% /data1
#
# time ./a.out bigfile 512
real 0m1.875s
user 0m0.000s
sys 0m0.730s
# du -h bigfile
513G bigfile
# rm -f bigfile
#
# ./a.out bigfile 512
# ls -l bigfile
-rw-r--r-- 1 root root 549755813888 Jul 1 09:48 bigfile

5 Responses to “Fault Injection Testing. Spurious Space Depletion? Sure, Why Not?”


  1. 1 goryszewskig July 4, 2012 at 12:21 pm

    Nice one! Time to try this on production to trigger new EMC storage buy. :)
    Regards
    GregG

    • 2 kevinclosson July 4, 2012 at 4:03 pm

      @goryszewskig : Thanks for stopping by! I have a kernel DLM that eats space even faster for those times I want to be nasty (fault-injection testing) but now that the fallocate() routine is prime time I promote its use for such purposes.

  2. 3 Simon Haslam August 25, 2012 at 2:18 am

    Sorry for being dense, but can’t you just dd a file from /dev/zero or /dev/random to fill up the filesystem artificially?

    • 4 Simon Haslam August 25, 2012 at 2:41 am

      Ah – sorry didn’t read your opening para carefully enough – I suppose dd is a bit CPU (for 1 core) and I/O intensive.

      My lab VMs tend to have quite small filesystems so as not to waste space so I find they inadvertently fill up quite often (e.g. I’ve seen clusterware produce 500MB or more of logs without too much effort). It’s not usually pretty…

    • 5 kevinclosson August 25, 2012 at 9:53 am

      Hi Simon,

      Yes you can but it is not ***immediate*** :-) That’s the evil aspect and when torture testing systems you have to be evil. So script a quick check for how much free space there is then fallocate() all but, say, 4K of it again and again in a loop in filesystems where Oracle (or any application) needs free space. It’s just a way to see whether spurious space depletion is a problem for your application to handle.


Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s




EMC Employee Disclaimer

The opinions and interests expressed on EMC employee blogs are the employees' own and do not necessarily represent EMC's positions, strategies or views. EMC makes no representation or warranties about employee blogs or the accuracy or reliability of such blogs. When you access employee blogs, even though they may contain the EMC logo and content regarding EMC products and services, employee blogs are independent of EMC and EMC does not control their content or operation. In addition, a link to a blog does not mean that EMC endorses that blog or has responsibility for its content or use.

This disclaimer was put into place on March 23, 2011.

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 2,145 other followers

Oracle ACE Program Status

Click It

website metrics

Fond Memories

Copyright

All content is © Kevin Closson and "Kevin Closson's Blog: Platforms, Databases, and Storage", 2006-2013. Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Kevin Closson and Kevin Closson's Blog: Platforms, Databases, and Storage with appropriate and specific direction to the original content.

Follow

Get every new post delivered to your Inbox.

Join 2,145 other followers

%d bloggers like this: