mm: retry more before OOM in the presence of slow shrinkers

Shrinkers like lowmemorykiller makes the decision of killing tasks
based on cached levels and also whether previous killed tasks have
been reaped. This can delay the reclaim and by this time no progress
loops of should_reclaim_retry can reach max retries, thus resulting in
premature kills. How fast the no progress loops can complete depends on
cpu speed, thus retry until reclaimable pages are too less (when
__zone_watermark_ok of shouuld_reclaim_retry fails) in the case of slow
shrinkers. This fixes premature order 0 OOMs.
Now, for (0 < order < costly order) cases, should_reclaim_retry can bail
out fast. The decision to try should_compact_retry is based on progress
made in the previous reclaim. A single no progress reclaim can result in
OOMs for higher orders. Fix that too.

Change-Id: I189311e25fc663a2e27d4dc4a38b24024c4cd2a3
Signed-off-by: Vinayak Menon <vinmenon@codeaurora.org>

mm/page_alloc.c

@@ -3793,7 +3793,8 @@ should_reclaim_retry(gfp_t gfp_mask, unsigned order,
 	 * their order will become available due to high fragmentation so
 	 * always increment the no progress counter for them
 	 */
-	if (did_some_progress && order <= PAGE_ALLOC_COSTLY_ORDER)
+	if ((did_some_progress && order <= PAGE_ALLOC_COSTLY_ORDER) ||
+			IS_ENABLED(CONFIG_HAVE_LOW_MEMORY_KILLER))
 		*no_progress_loops = 0;
 	else
 		(*no_progress_loops)++;
@@ -4095,7 +4096,8 @@ retry:
 	 * implementation of the compaction depends on the sufficient amount
 	 * of free memory (see __compaction_suitable)
 	 */
-	if (did_some_progress > 0 &&
+	if ((did_some_progress > 0 ||
+			IS_ENABLED(CONFIG_HAVE_LOW_MEMORY_KILLER)) &&
 			should_compact_retry(ac, order, alloc_flags,
 				compact_result, &compact_priority,
 				&compaction_retries))