> I really wonder if a "simple" memory-aware scheduler that punished tasks whose memory behavior (allocation or access) slows down the system would be enough. What if such a process was punished (for slowing down the system) by being slowed down (not being scheduled or getting lower cpu times) in a way that, no matter what it did, the other tasks continued fast enough so that it could be (even manually) killed without soft-crashing the system?
This approach is hard to make work, because once the system is in memory shortage, mostly all processes will be slowing the system. There's already a penalty for accessing memory that's not currently paged in --- the process will be descheduled pending the i/o, and other processes can run during that time ... until they access memory that's not paged in. You can easily get into a situation where most of your cpu time is spend in paging and no useful work gets done. This can happen even without swap; the paging will just happen on memory mapped files, even if you're not using mmap for data files, your executables and libraries are mmaped, so the system will page those out and in in an effort to manage the memory shortage.
To make a system easier to operate, I like to run with a small swap partition and monitor swap usage both in % and by rate. You can often get a small window of a still responsive system to try to identify the rogue process and kill it without having to restart the whole thing. A small partition means a big problem will quickly hit the OOM killer without being in swap hell for ages.
There might be research or practice from commercial unix and mainframe where multi-tenancy is more common? What I've seen on the free software side is mostly avoiding the issue or trying to addressing it with policy limits on memory usage. Probably more thorough memory accounting is a necessary step to doing a better job, but adding more ram when you run into problems is effective mitigation, so....
This approach is hard to make work, because once the system is in memory shortage, mostly all processes will be slowing the system. There's already a penalty for accessing memory that's not currently paged in --- the process will be descheduled pending the i/o, and other processes can run during that time ... until they access memory that's not paged in. You can easily get into a situation where most of your cpu time is spend in paging and no useful work gets done. This can happen even without swap; the paging will just happen on memory mapped files, even if you're not using mmap for data files, your executables and libraries are mmaped, so the system will page those out and in in an effort to manage the memory shortage.
To make a system easier to operate, I like to run with a small swap partition and monitor swap usage both in % and by rate. You can often get a small window of a still responsive system to try to identify the rogue process and kill it without having to restart the whole thing. A small partition means a big problem will quickly hit the OOM killer without being in swap hell for ages.
There might be research or practice from commercial unix and mainframe where multi-tenancy is more common? What I've seen on the free software side is mostly avoiding the issue or trying to addressing it with policy limits on memory usage. Probably more thorough memory accounting is a necessary step to doing a better job, but adding more ram when you run into problems is effective mitigation, so....