Little Things Doth Crabby Make – Part XVI. Hey ls(1) And du(1) Are Supposed To Agree.

BLOG UPDATE 08-NOV-2011 : After posting and handling comments I realize the title is drawing readers’ attention to the wrong thing. I am aware that ls(1) and du(1) can and should disagree about file size and space utilization because they are two different things (e.g., sparse files and non-file data associated with the file). The point of this blog was just to show that within microseconds (or less) ls(1) reported my file was zero bytes and du(1) reported it occupied nearly 300GB space in the file system. That is the “oddity”–or is it? The original post follows:

I love XFS.

I’ve been doing some, shall we say, rather unseemly sorts of things to one of my XFS file systems. Things are generally holding up, however, a little something made me crabby today so I think another installment in my Little Things Doth Crabby Make series is in order.

I don’t think I even need to explain why the following is something which hath crabby made:

# uname -r
2.6.18-194.26.1.el5
# df -h .
Filesystem            Size  Used Avail Use% Mounted on
/dev/sdb2             2.0T  1.7T  236G  88% /data1
# ls -l foo
-rw-r--r-- 1 root root 0 Nov  7 10:53 foo
# du -sh foo
287G    foo
# rm -f foo
$ ls -l foo
ls: foo: No such file or directory
# df -h .
Filesystem            Size  Used Avail Use% Mounted on
/dev/sdb2             2.0T  1.7T  236G  88% /data1

I’ll blog more on what’s happening here as soon as I can. In the meantime, I’m crabby.  Not really, that’s just the theme of this blog series.

11 Responses to “Little Things Doth Crabby Make – Part XVI. Hey ls(1) And du(1) Are Supposed To Agree.”


  1. 1 Paul Janda November 7, 2011 at 8:57 pm

    We like XFS, too. I politely asked the s/a about something similar. To reuse your commentary, for the purpose of paraphrasing the response I got, the s/a said he was going to do some unseemly things to me if I didn’t find something better to do. That doth crabby me make.

  2. 2 kevinclosson November 7, 2011 at 9:14 pm

    Hi Paul,

    That’s funny! Hmmm…I noticed your email when I approved your comment. I wonder if your s/a has been in your IT shop long enough to remember my on-site (at your data center) days back in the very early 1990s when you folks were one of the very first production sites to go live with Oracle Parallel Server on Unix (Sequent)? Ah, memories.

    That aside, may I ask what sort of thing you were doing when you discovered a discrepancy between du and ls opinions of a file’s size? I will blog what got me to this point soon enough, but curious about your situation too.

  3. 3 Mark W. Farnham November 7, 2011 at 9:33 pm

    If ls and du report the same how am I ‘sposed ta know someone still has a deleted file open?

    They make powder for that crabby feeling Kevin.

    I look forward to your blog series!

  4. 4 Piavlo November 7, 2011 at 10:15 pm

    Might be a classic case that the foo file was unlinked from the file system put some process still holds fd to this file – so you see no df difference before and after the rm command? But the fact that it showed 0 file size – might simply indicate that the inode is corrupted. Did you try to run “xfs_bmap -l foo”? Did you try to copy this file to see if the copy is also of 0 size but du would still show 287G ?

  5. 6 kevinclosson November 7, 2011 at 10:24 pm

    Mark,

    Would that be gunpowder? 🙂

    You *do* know, don’t you, that having a file descriptor doesn’t keep the file in the namespace. That aside, pray-tell how you suppose “someone” with an active file descriptor is going to make it so that immediate (scripted), back-to-back ls(1) and du(1) commands are going to differ in their reckoning of how large the file is? Ls(1) tells me zero bytes, du(1) says 287G.

    As for the open-but-deleted scenario:

    # ls -l foo
    -rw-r–r– 1 root root 0 Nov 7 13:58 foo
    # ls -l foo
    -rw-r–r– 1 root root 0 Nov 7 13:58 foo
    #
    # rm foo
    # ls -l foo
    ls: foo: No such file or directory
    # pwd
    /tmp
    # ls -l /proc/22515/fd
    total 0
    lrwx—— 1 root root 64 Nov 7 13:57 0 -> /dev/pts/14
    lrwx—— 1 root root 64 Nov 7 13:57 1 -> /dev/pts/14
    lrwx—— 1 root root 64 Nov 7 13:57 2 -> /dev/pts/14
    lrwx—— 1 root root 64 Nov 7 13:57 3 -> /tmp/foo (deleted)
    #
    # echo $$
    22442

  6. 7 jgarry November 7, 2011 at 11:03 pm

    ls reports the filesize, du reports the blocks allocated to the inode. They’re not supposed to always agree. With the ability to speculatively allocate, there could easily be a reasonable delay or a bug with du.

    • 8 kevinclosson November 8, 2011 at 12:17 am

      jgarry,

      So why then is it that ls thinks its zero bytes and du sees hundreds of gigabytes of allocated space? Not sparse but allocated space? Your explanation works for huge ls/small-du but not the other way around.

      Also… I’m not sure I understand your point about ls and du semantic differences. On linux they both simply stat() the file.

      • 9 jgarry November 8, 2011 at 10:43 pm

        Yes, I have no idea what I am talking about, this is entirely speculative. See http://permalink.gmane.org/gmane.comp.file-systems.xfs.general/36258
        So what I am trying to say is, ls is correct, and du is not misleading because xfs has not deallocated the space yet, blocks are still pending to the inode. They potentially only get dropped when the last reference to the inode is dropped. Yes, I know I am going way out on a limb speculating from speculative preallocation to post-deallocation and assuming they both don’t simply stat() the file. But that’s ok because I have no idea what I am talking about. That way I don’t have to be crabby.

        • 10 kevinclosson November 8, 2011 at 11:04 pm

          Joel wrote: “and assuming they both don’t simply stat() the file.”

          …Joel,

          You are the one saying you don’t know what you are talking about. Not me.

          I specifically pointed out that both ls and du do, indeed, stat the file.

          I also pointed out that I was doing “unseemly sorts of things” to my file system. I’ll explain soon.

          Come to think of it, I’m probably throwing readers for a loop with the title of the post. Of course ls can report much larger numbers than du in sparse cases but since du proved it isn’t sparse that point is moot.


  1. 1 Little Things Doth Crabby Make – Part XVI (Addendum). Hey ls(1) And du(1) Are Supposed To Agree. « Ukrainian Oracle User Group Trackback on November 23, 2011 at 2:31 am

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s




DISCLAIMER

I work for Amazon Web Services. The opinions I share in this blog are my own. I'm *not* communicating as a spokesperson for Amazon. In other words, I work at Amazon, but this is my own opinion.

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 2,974 other followers

Oracle ACE Program Status

Click It

website metrics

Fond Memories

Copyright

All content is © Kevin Closson and "Kevin Closson's Blog: Platforms, Databases, and Storage", 2006-2015. Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Kevin Closson and Kevin Closson's Blog: Platforms, Databases, and Storage with appropriate and specific direction to the original content.

%d bloggers like this: