Accessing metrics and logs on your own analytics platform

This page explains how to view SpatialOS metrics within your own in-house or third-party analytics platform, using the monitoring proxy that lets you access your deployment's metrics.

You can also use the monitoring proxy to access metrics through code.

In your analytics platform, you'll need to:

  1. Add Prometheus and Elasticsearch as data sources.
  2. Add the monitoring proxy URL.
  3. Enter the HTTP basic authentication username and password described below.

In general

  • Follow best practice for the solutions you are using.

  • Sending the password

  • Use client-side encryption software such as GPG, or share the password via LastPass.
  • Make sure the recipient is who they claim to be: use a system with strong two-factor authentication or rigorous key-agreement protocols.
  • Make sure recipients are aware of the sensitivity of the password.

  • Storing the password

  • Do not store the password in plaintext on machines. Use a cryptographically secure secret management system like LastPass, Vault, or GPG to save a backup of the key.
  • Require full-disk encryption on machines that handle the password.
  • Securely wipe the password from machines when it is no longer needed.

  • Using the password

  • Make sure the analytics platform daemon runs as its own user, and no other unprivileged users on the system have permission to access the password.
  • Control physical access to the server.
  • Restrict administrative access to the server to a small and well-documented set of people.
  • Use reputable and securely configured servers for admin access (for example, using SSH key-based authentication).
  • Make sure the machine that hosts the local analytics platform instance does not expose endpoints unnecessarily, especially to the Internet.
  • Make sure the system does not run other services unless they are segregated into an isolated security context.
  • We've provided a worked example below using Grafana.

    Authentication

    Make sure you use HTTPS when making requests against the monitoring proxy.

    The monitoring proxy uses HTTP basic authentication. You will need to supply longshot_user as the username, and a valid refresh token as the password.

    For access from a dashboard or other automated service, we strongly recommend you use the refresh token of a service account. To set up a service account, please reach out to Improbable customer support via the helpdesk (for customers with a service agreement) or the forums.

    Worked example: Grafana

    Before you start

    • If you don't already have a Grafana server, you'll need to set one up. You can install one yourself or use Grafana cloud.

    • This service is in alpha, so is subject to change. In particular, all URLs (especially
      https://monitoring.service.improbable.io/@proxyhost/) and metric names are
      likely to change in the future.

    Set up SpatialOS metrics in your own Grafana instance

    1. Log in to your Grafana server as an admin.
    2. Under Data Sources, add a new Prometheus Data Source:
    Data source fields Details
    Data source name SpatialOS
    Data source type Prometheus
    URL <copy this link>
    HTTP access Proxy
    HTTP auth Basic auth
    Basic auth details The username and password received from Improbable customer support
    1. Now you can add graphs of these metrics to your dashboard.

      For details of how to do this, see the Grafana docs on Creating a Prometheus graph.

      You can see which metrics are available, and get example queries to run, on the Metrics reference page.

    Only create alerts using documented metrics.

    Be careful with the queries you write: if they're very expensive, you may experience problems with the metrics servers. If you're not sure, ask customer support to review the queries.

    Don't set a refresh rate more frequent than 60 seconds: if the refresh rate is too frequent, requests from your dashboard may fail due to the restrictions on metric querying.

    Restrictions on metric querying

    • Queries must specify the project being queried. For example:
        spatialos_worker_connected::sum{project="test_project"}
    
    • Each project has a limit of 200 queries per minute.
      For example, if one user issues 200 of the above queries, no users will be able to query metrics for that project for up to a minute.
    • Each project has a limit of 60s of query processing time, reset every minute.
    • Queries will time out after 30s of processing.

    2019-09-23 Page updated with editorial review: Removed Set up SpatialOS logs in your own Grafana instance

    Updated about a year ago


    Accessing metrics and logs on your own analytics platform


    Suggested Edits are limited on API Reference Pages

    You can only suggest edits to Markdown body content, but not to the API spec.