Comment 33 for bug 1833281

On Thu 24-08-17 00:30:40, Steven Haigh wrote:
> On Wednesday, 23 August 2017 11:38:48 PM AEST Michal Hocko wrote:
> > On Tue 22-08-17 15:55:30, Andrew Morton wrote:
> > > (switched to email. Please respond via emailed reply-to-all, not via the
> > > bugzilla web interface).
> >
> > > On Tue, 22 Aug 2017 11:17:08 +0000 <email address hidden>
> wrote:
> > [...]
> >
> > > Sadly I haven't been able to capture this information
> > >
> > > > fully yet due to said unresponsiveness.
> >
> > Please try to collect /proc/vmstat in the bacground and provide the
> > collected data. Something like
> >
> > while true
> > do
> > cp /proc/vmstat > vmstat.$(date +%s)
> > sleep 1s
> > done
> >
> > If the system turns out so busy that it won't be able to fork a process
> > or write the output (which you will see by checking timestamps of files
> > and looking for holes) then you can try the attached proggy
> > ./read_vmstat output_file timeout output_size
> >
> > Note you might need to increase the mlock rlimit to lock everything into
> > memory.
>
> Thanks Michal,
>
> I have upgraded PCs since I initially put together this data - however I was
> able to get strange behaviour by pulling out an 8Gb RAM stick in my new
> system
> - leaving it with only 8Gb of RAM.
>
> All these tests are performed with Fedora 26 and kernel
> 4.12.8-300.fc26.x86_64
>
> I have attached 3 files with output.
>
> 8Gb-noswap.tar.gz contains the output of /proc/vmstat running on 8Gb of RAM
> with no swap. Under this scenario, I was expecting the OOM reaper to just
> kill
> the game when memory allocated became too high for the amount of physical
> RAM.
> Interestingly, you'll notice a massive hang in the output before the game is
> terminated. I didn't see this before.

I have checked few gaps. E.g. vmstat.1503496391 vmstat.1503496451 which
is one minute. The most notable thing is that there are only very few
pagecache pages
   [base] [diff]
nr_active_file 1641 3345
nr_inactive_file 1630 4787

So there is not much to reclaim without swap. The more important thing
is that we keep reclaiming and refaulting that memory

workingset_activate 5905591 1616391
workingset_refault 33412538 10302135
pgactivate 42279686 13219593
pgdeactivate 48175757 14833350

pgscan_kswapd 379431778 126407849
pgsteal_kswapd 49751559 13322930

so we are effectivelly trashing over the very small amount of
reclaimable memory. This is something that we cannot detect right now.
It is even questionable whether the OOM killer would be an appropriate
action. Your system has recovered and then it is always hard to decide
whether a disruptive action is more appropriate. One minute of
unresponsiveness is certainly annoying though. Your system is obviously
under provisioned to load you want to run obviously.

It is quite interesting to see that we do not really have too many
direct reclaimers during this time period
allocstall_normal 30 1
allocstall_movable 490 88
pgscan_direct_throttle 0 0
pgsteal_direct 24434 4069
pgscan_direct 38678 5868

> 8Gb-swap-on-file.tar.gz contains the output of /proc/vmstat still with 8Gb of
> RAM - but creating a file with swap on the PCIe SSD /swapfile with size 8Gb
> via:
> # dd if=/dev/zero of=/swapfile bs=1G count=8
> # mkswap /swapfile
> # swapon /swapfile
>
> Some times (all in UTC+10):
> 23:58:30 - Start loading the saved game
> 23:59:38 - Load ok, all running fine
> 00:00:15 - Load Chrome
> 00:01:00 - Quit the game
>
> The game seemed to run ok with no real issue - and a lot was swapped to the
> swap file. I'm wondering if it was purely the speed of the PCIe SSD that
> caused this appearance - as the creation of the file with dd completed at
> ~1.4GB/sec.

Swap IO tends to be really scattered and the IO performance is not really
great even on a fast storage AFAIK.

Anyway your original report sounded like a regression. Were you able to
run the _same_ workload on an older kernel without these issues?