Maximize RLIMIT_AS resource limit in nacl_helper child before exec

NaCl processes need an unusually large amount of address space, because
they do huge PROT_NONE mappings to reserve contiguous regions much larger
than the memory that will actually be used.  An ambient RLIMIT_AS setting
can break this, so we need to raise the soft limit to the hard limit (which
is usually unlimited).

We cannot rely on doing this inside the nacl_helper child itself, so we
must do it in the zygote before it exec's nacl_helper_bootstrap.  That exec
itself could fail if the limit is too small, because nacl_helper_bootstrap
uses ELF program headers to reserve a large region of address space.

BUG= http://code.google.com/p/nativeclient/issues/detail?id=2438
TEST= nacl apps work when "ulimit -S -v 5376320" was run before starting chrome

R=mark@chromium.org,bsy@google.com,bradchen@google.com

Review URL: http://codereview.chromium.org/8528041

git-svn-id: svn://svn.chromium.org/chrome/trunk/src@110123 0039d316-1c4b-4281-b951-d872f2087c98

1 files changed, 12 insertions, 0 deletions

diff --git a/chrome/app/nacl_fork_delegate_linux.cc b/chrome/app/nacl_fork_delegate_linux.cc
index 372a5b7..937f690 100644
--- a/chrome/app/nacl_fork_delegate_linux.cc
+++ b/chrome/app/nacl_fork_delegate_linux.cc
@@ -6,6 +6,7 @@
 
 #include <signal.h>
 #include <stdlib.h>
+#include <sys/resource.h>
 #include <sys/socket.h>
 
 #include "base/basictypes.h"
@@ -68,6 +69,17 @@ void NaClForkDelegate::Init(const bool sandboxed,
     base::LaunchOptions options;
     options.fds_to_remap = &fds_to_map;
     options.clone_flags = CLONE_FS | SIGCHLD;
+
+    // The NaCl processes spawned may need to exceed the ambient soft limit
+    // on RLIMIT_AS to allocate the untrusted address space and its guard
+    // regions.  The nacl_helper itself cannot just raise its own limit,
+    // because the existing limit may prevent the initial exec of
+    // nacl_helper_bootstrap from succeeding, with its large address space
+    // reservation.
+    std::set<int> max_these_limits;
+    max_these_limits.insert(RLIMIT_AS);
+    options.maximize_rlimits = &max_these_limits;
+
     if (!base::LaunchProcess(cmd_line.argv(), options, NULL))
       status_ = kNaClHelperLaunchFailed;
     // parent and error cases are handled below