...
This shows that for system_call_fastpath the majority of the time is in ll_intent_file_open (work from both to top of the call graph, each higher entry's percentage is the sum of all predecessors plus itself). All of those calls are part of the Lustre kernel. Time spent processing the kernel threads is charged as system time, threads that lock the core waiting for response like ll_file_open likely get charged as system time rather than idle time. Threads that lock the core because they're inherently intended to wait, like spin_lock, are likely charged as idle time. If true that explains the increased system time in addition to idle time. It would be better if the kernel had a 5th state to account for enforced system waits rather than mixing them between system time and idle time. That's what you get with a free OS/kernel I guess .
Lastly looking at the ganglia plots for the Metadata server (MDS) for the last year one can see a sharp increase in system time that began in October, roughly consistent with the cluster upgrade.
...