tools/android/loading Initial infrastructure for trace collection on GCE

This CL adds some configuration and a http server in prevision to
running the trace collection on GCE.
It is far from complete, and does not collect any traces yet.

Review URL: https://codereview.chromium.org/1777523002

Cr-Commit-Position: refs/heads/master@{#380404}

4 files changed, 279 insertions, 0 deletions

diff --git a/tools/android/loading/gce/README.md b/tools/android/loading/gce/README.md
new file mode 100644
index 0000000..eda5520
--- /dev/null
+++ b/tools/android/loading/gce/README.md
@@ -0,0 +1,132 @@
+# Clovis in the Cloud: Developer Guide
+
+This document describes how to collect Chromium traces using Google Compute
+Engine.
+
+[TOC]
+
+## Initial setup
+
+Install the [gcloud command line tool][1].
+
+Checkout the source:
+
+```shell
+mkdir clovis
+cd clovis
+gcloud init
+```
+
+When offered, accept to clone the Google Cloud repo.
+
+## Update or Change the code
+
+Make changes to the code, or simply copy the latest version from Chromium into
+your local Google Cloud repository. Then commit and push:
+
+```shell
+git commit
+git push -u origin master
+```
+
+If there are instances already running, they need to be restarted for this to
+take effect.
+
+## Start the app in the cloud
+
+Create an instance using latest ubuntu LTS:
+
+```shell
+gcloud compute instances create clovis-tracer-1 \
+ --machine-type n1-standard-1 \
+ --image ubuntu-14-04 \
+ --zone europe-west1-c \
+ --tags clovis-http-server \
+ --scopes cloud-platform \
+ --metadata-from-file startup-script=default/startup-script.sh
+```
+
+This should output the IP address of the instance.
+Otherwise the IP address can be retrieved by doing:
+
+```shell
+gcloud compute instances list
+```
+
+Interact with the app on the port 8080 in your browser at
+`http://<instance-ip>:8080`.
+
+TODO: allow starting the instance in the cloud without Supervisor. This enables
+iterative development on the instance using SSH, manually starting and stopping
+the app. This can be done using [instance metadata][2].
+
+## Stop the app in the cloud
+
+```shell
+gcloud compute instances delete clovis-tracer-1
+```
+
+## Connect to the instance with SSH
+
+```shell
+gcloud compute ssh clovis-tracer-1
+```
+
+## Use the app locally
+
+Setup the local environment:
+
+```shell
+virtualenv env
+source env/bin/activate
+pip install -r pip_requirements.txt
+```
+
+Launch the app:
+
+```shell
+gunicorn --workers=2 main:app --bind 127.0.0.1:8000
+```
+
+In your browser, go to `http://localhost:8000` and use the app.
+
+Tear down the local environment:
+
+```shell
+deactivate
+```
+
+## Project-wide settings
+
+This is already setup, no need to do this again.
+Kept here for reference.
+
+### Server configuration file
+
+`main.py` expects to find a `server_config.json` file, which is a dictionary
+with the keys:
+
+*   `project_name`: the name of the Google Compute project,
+*   `bucket_name`: the name of the Google Storage bucket used to store the
+    results.
+
+### Firewall rule
+
+Firewall rule to allow access to the instance HTTP server from the outside:
+
+```shell
+gcloud compute firewall-rules create default-allow-http-8080 \
+    --allow tcp:8080 \
+    --source-ranges 0.0.0.0/0 \
+    --target-tags clovis-http-server \
+    --description "Allow port 8080 access to http-server"
+```
+
+The firewall rule can be disabled with:
+
+```shell
+gcloud compute firewall-rules delete default-allow-http-8080
+```
+
+[1]: https://cloud.google.com/sdk
+[2]: https://cloud.google.com/compute/docs/startupscript#custom
diff --git a/tools/android/loading/gce/main.py b/tools/android/loading/gce/main.py
new file mode 100644
index 0000000..67fe3d1
--- /dev/null
+++ b/tools/android/loading/gce/main.py
@@ -0,0 +1,87 @@
+# Copyright 2016 The Chromium Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+import json
+
+from gcloud import storage
+from gcloud.exceptions import NotFound
+from oauth2client.client import GoogleCredentials
+
+class ServerApp(object):
+  """Simple web server application, collecting traces and writing them in
+  Google Cloud Storage.
+  """
+
+  def __init__(self):
+    print 'Initializing credentials'
+    self._credentials = GoogleCredentials.get_application_default()
+    print 'Reading server configuration'
+    with open('server_config.json') as configuration_file:
+       self._config = json.load(configuration_file)
+
+  def _GetStorageClient(self):
+    return storage.Client(project = self._config['project_name'],
+                          credentials = self._credentials)
+
+  def _GetStorageBucket(self, storage_client):
+    return storage_client.get_bucket(self._config['bucket_name'])
+
+  def _UploadFile(self, file_stream, filename):
+    client = self._GetStorageClient()
+    bucket = self._GetStorageBucket(client)
+    blob = bucket.blob(filename)
+    blob.upload_from_string(file_stream)
+    url = blob.public_url
+    return url
+
+  def _DeleteFile(self, filename):
+    client = self._GetStorageClient()
+    bucket = self._GetStorageBucket(client)
+    try:
+      bucket.delete_blob(filename)
+      return True
+    except NotFound:
+      return False
+
+  def _ReadFile(self, filename):
+    client = self._GetStorageClient()
+    bucket = self._GetStorageBucket(client)
+    blob = bucket.get_blob(filename)
+    if not blob:
+      return None
+    return blob.download_as_string()
+
+  def __call__(self, environ, start_response):
+    path = environ['PATH_INFO']
+    if path == '/favicon.ico':
+        start_response('404 NOT FOUND', [('Content-Length', '0')])
+        return iter([''])
+
+    status = '200 OK'
+
+    if path == '/write':
+      url = self._UploadFile('foo', 'test.txt')
+      data = 'Writing file at\n' + url + '\n'
+    elif path == '/read':
+      data = self._ReadFile('test.txt')
+      if not data:
+        data = ''
+        status = '404 NOT FOUND'
+    elif path == '/delete':
+      if self._DeleteFile('test.txt'):
+        data = 'Success\n'
+      else:
+        data = 'Failed\n'
+    else:
+      data = environ['PATH_INFO'] + '\n'
+
+    response_headers = [
+        ('Content-type','text/plain'),
+        ('Content-Length', str(len(data)))
+    ]
+    start_response(status, response_headers)
+    return iter([data])
+
+
+app = ServerApp()
diff --git a/tools/android/loading/gce/pip_requirements.txt b/tools/android/loading/gce/pip_requirements.txt
new file mode 100644
index 0000000..e2d68b7
--- /dev/null
+++ b/tools/android/loading/gce/pip_requirements.txt
@@ -0,0 +1,2 @@
+gunicorn==19.4.5
+gcloud==0.10.1
diff --git a/tools/android/loading/gce/startup-script.sh b/tools/android/loading/gce/startup-script.sh
new file mode 100644
index 0000000..4f2cafa
--- /dev/null
+++ b/tools/android/loading/gce/startup-script.sh
@@ -0,0 +1,58 @@
+# Copyright 2016 The Chromium Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+# Script executed at instance startup. It installs the required dependencies,
+# downloads the source code, and starts a web server.
+
+set -v
+
+# Talk to the metadata server to get the project id
+PROJECTID=$(curl -s \
+    "http://metadata.google.internal/computeMetadata/v1/project/project-id" \
+    -H "Metadata-Flavor: Google")
+
+# Install dependencies from apt
+apt-get update
+apt-get install -yq git supervisor python-pip python-dev libffi-dev libssl-dev
+
+# Create a pythonapp user. The application will run as this user.
+useradd -m -d /home/pythonapp pythonapp
+
+# pip from apt is out of date, so make it update itself and install virtualenv.
+pip install --upgrade pip virtualenv
+
+# Get the source code from the Google Cloud Repository
+# git requires $HOME and it's not set during the startup script.
+export HOME=/root
+git config --global credential.helper gcloud.sh
+git clone https://source.developers.google.com/p/$PROJECTID /opt/app/clovis
+
+# Install app dependencies
+virtualenv /opt/app/clovis/env
+/opt/app/clovis/env/bin/pip install -r /opt/app/clovis/pip_requirements.txt
+
+# Make sure the pythonapp user owns the application code
+chown -R pythonapp:pythonapp /opt/app
+
+# Configure supervisor to start gunicorn inside of our virtualenv and run the
+# applicaiton.
+cat >/etc/supervisor/conf.d/python-app.conf << EOF
+[program:pythonapp]
+directory=/opt/app/clovis
+command=/opt/app/clovis/env/bin/gunicorn --workers=2 main:app \
+  --bind 0.0.0.0:8080
+autostart=true
+autorestart=true
+user=pythonapp
+# Environment variables ensure that the application runs inside of the
+# configured virtualenv.
+environment=VIRTUAL_ENV="/opt/app/env/clovis",PATH="/opt/app/clovis/env/bin",\
+    HOME="/home/pythonapp",USER="pythonapp"
+stdout_logfile=syslog
+stderr_logfile=syslog
+EOF
+
+supervisorctl reread
+supervisorctl update
+