Kevin Closson's Blog: Platforms, Databases and Storage

20 Responses to “Linux is Perfect So Why Would You Monitor Performance?”

1. 1 Marco Gralike October 19, 2008 at 1:57 pm

   Thanks for the tip. Any comments about why it is better then, for example, using NMON ?

   Reply
2. 2 Marco Gralike October 19, 2008 at 2:09 pm

   Forget it 😉

   As said, thanks for the tip and forget the stupid question about nmon.

   🙂

   Reply
3. 3 Jakub Wartak October 20, 2008 at 2:51 pm

   Kevin, do you have any clue/idea how to monitor on Linux
   1) read(2)/write(2) avg syscall time per running process (I/O completition, or lets better say to VM – yup, without O_DIRECT)
   2) how to measure seqential vs random ratio for process/disk ?
   3) VM ratio hit?

   E.g. i could use something similliar to “strace -r -s 0 -e read,write -p ” for point 1 but would like it to being less intrusive.

   For second, is there is any other option on Linux that writing dedicated ptrace(2)-based utility tracing lseek(2),read(2),write(2) ??

   3) Any idea??????

   Reply
4. 4 kevinclosson October 20, 2008 at 3:12 pm

   Jakub,

   Perhaps you might investigate OProfile? Strace single-steps processes so it is worthless in this regard.

   Reply
5. 5 Mark Seger October 20, 2008 at 6:48 pm

   Jakub – This probably isn’t exactly what you’re looking for, but a feature of collectl that I don’t think has gotten a lot of use its ability to show I/O by process, assuming of course you have a kernel > 2.6.22 that was also built with process accounting enabled. See http://collectl.sourceforge.net/Process.html for more details.

   The reason I bring it up is you’ll at least be able to see a processes I/O rates as well as the number of I/O systems calls its doing if you select the I/O summary format.

   And of course you can see it at the individual thread level as well if you so choose by including –procopts t.

   -mark

   Reply
6. 6 tanelp October 21, 2008 at 9:25 am

   Kevin,

   Regarding single-stepping, isn’t the strace system call tracing done via trapping the syscall entry and return only and not via single-stepping the whole execution path?

   Tanel.

   Reply
7. 7 tanelp October 21, 2008 at 10:20 am

   …I’m talking about ptrace() PTRACE_SYSCALL option here…

   Reply
8. 8 kevinclosson October 21, 2008 at 1:39 pm

   Tanel, I think you are right. I wonder if my wires were crossed between it and truss (or if even truss does that)…hmmm…I’ll have to investigate and clarify. I’ve got single-stepping in my mind somehow…need to figure out why 🙂

   Reply
9. 9 tanelp October 21, 2008 at 8:34 pm

   You’ve been debugging too much lately? 😉

   I think truss doesn’t do single-stepping either, at least in Solaris, as it uses /proc/pid/ctl interface which supports setting breakpoint traps at specific system calls (there are ctl flags PCSENTRY and PCSEXIT).

   Nevertheless, stracing / trussing critical (background) processes in production should require careful thought first as you can cause trouble to traced process (missed signals, failed IOs) as I’ve once evidenced here: http://blog.tanelpoder.com/2008/06/14/debugger-dangers/

   Jakub, in addition to OProfile, if you’re on recent Linux (RHEL 5+ equivalent) then you can look into SystemTap. It is getting mature enough now for professional (non-hacker) use – on my CentOS 5.2 install it’s kernel and userland parts are preinstalled so I didn’t need to start patching and recompiling kernel (for which I just don’t have the time and will).

   Reply
10. 10 kevinclosson October 21, 2008 at 9:30 pm

    I’ve just been doing too much. Period.

    Reply
11. 11 Mdx4 October 27, 2008 at 3:28 pm

    Why don’t use SAR that is standard in all unices and well know to all system admins ?

    Reply
12. 12 Mark Seger October 31, 2008 at 5:30 pm

    mdx4 – I guess you still don’t get it and maybe won’t until you try collectl out and see for yourself. Can sar produce data in a format you can easily plot? Can sar collect data on infiniband? slabs? and other ‘less common but still critical’ types of data? can sar run at sub-second monitoring levels? can sar provide much of its output on a single line for easy reading? can you easily feed sar data to other programs over a socket in real-time? far too many other things to list. I think sar is fine at what it does, but there is so much more it doesn’t do and that’s why I wrote collectl.
    -mark

    Reply
13. 13 Jakub Wartak November 6, 2008 at 8:34 pm

    Ok, thanks for valuable hints. I think that SystemTap is the long time solution, but I wanted to look at old systems too. It even seems that Oracle wrote some part of public SystemTap scripts 😉

    I’ve started developing ptrace(2) (lseek/read/write, pwrite is currently missing) based fileops aggregation utility to trace I/O characteristic on fds , when I’ve hit strange issues in production (as Tanel mentioned) for Java apps, one even DoSed my whole system. Maybe sometime I’ll have time to finish & release it.

    However I still get no idea how to calculate VM ratio hit over time…

    Reply
14. 14 Mark Seger March 14, 2009 at 12:31 pm

    I recently discovered /proc/buddyinfo, which shows how memory is fragmented, thought it useful and added it to collectly so now you can literally see in real-time how many fragments of what sizes are available. However I don’t have a real good feel for how much extra overhead is being generated by having too many small fragments and not enough larger one. Could it make a bigger difference for heavy data movers like oracle? Is this new look at memory fragmentation of use to anyone?

    -mark

    Reply
15. 15 Mark Seger April 27, 2009 at 8:29 pm

    I wasn’t sure where to post this to I thought I’d tack it onto the end of this string.

    I just released a new version of collectl that might even get Kevin’s attention because it now has a data collectl API. This means if there are any readily available Oracle specific stats, it shouldn’t be too tough (if you’re a perl programmer) to add them into collectl.

    Why would you want to do this? That’s easy – imagine looking at some oracle data and cpu/disk/interrupt/ctx/etc data ALL at the same time either on your terminal or maybe even in a plot – yes, this mechanism allows your custom data to be recorded in plot format too.

    You’re seeing Oracle I/O rates drop? Now you can watch linux stats and see if any resources are starved and causing your problems. Of course you could still do that without this feature but just think how much harder it would be.

    -mark

    Reply
    • 16 kevinclosson April 27, 2009 at 9:47 pm

      Hi Mark,

      Well, you know I’m a huge fan…didn’t know if you were aware of this or not, but collectl made the grade for inclusion into Exadata packaging. You really produced a winner with collectl!

      Reply
16. 17 Mark Seger October 18, 2009 at 1:27 pm

    Time for another update – a couple of weeks ago I open-sourced a set of utilities I had written awhile back for collectl, called collectl-utils. There’s a pretty good writeup at http://collectl-utils.sourceforge.net/ so I won’t repeat myself with what’s already stated there, other than to say there’s a very cool [my opinion] web-based plotting utility and another that makes it real easy to watch multiple systems at once. I even have a picture of watching the cpu load on 192 systems once a second, all displayed on a single very wide line (takes 3 monitors)!

    This could also morph into a related discussion on visualization of data and monitoring intervals. I’m alway amazed how many people run sar at the default monitoring interval of 10 minutes and think they’re ‘protected’ because in the event of a failure they’ll know what went wrong. Even ganglia users tend to use coarser intervals of a minute or more and may not even realize they’re not seeing ‘the whole truth’ about what their systems are doing.

    I’m a fan of 1-10 second intervals because they give you a ton of data and but you almost always have to plot it unless you know an exact time period to drill into. That’s why I made generating output as plottable data a key feature of collectl and also why I wrote colplot. Having tried a number of plotting tools I continue to be impressed with what a great job gnuplot does in terms of speed and detail. It’s plots may not be as ‘pretty’ as excel or ganglia but I think accuracy trumps pretty, which is something not all plotting packages can claim.

    -mark

    Reply
    • 18 kevinclosson October 18, 2009 at 5:03 pm

      Hi Mark,

      Love the kit! Updates always welcome here!

      Reply
17. 19 Mark Seger February 22, 2011 at 3:51 pm

    Hey Kevin – it’s been awhile but the wait has been worth it!

    I’ve just rewritten my colmux tool, part of collectl-utils and if you haven’t tried it before I promise that with this rewrite you’ll never put it down. It’s always been able to monitor clusters of nodes by multiplexing collectl output from multiple systems into a single display, but I’ve now added the ability to sort on any column and present the data in a top-like manner.

    The results are stunning! You can now look at virtually anything collectl can report across your entire cluster! Want to find a slow disk out of thousands? Piece of cake. Looking for memory leaks – just sort the process output by memory. How about an oracle process run amok – I haven’t tried that but I’m sure you will. 😉

    If you want to see what I posted about it on sourceforge you can either look it up there or just click on this – http://collectl-utils.sourceforge.net/colmux.html

    enjoy…

    -mark

    Reply
18. 20 kevinclosson February 22, 2011 at 6:55 pm

    Hi Mark,

    Great news. Thanks for the update and, of course, for the great work on collectl.

    Reply