Raw worker logs


Raw worker logs provide a simple, lightweight method to output unstructured logs from worker code and inspect the results. Unlike normal deployment logs, these logs are not sent through the SpatialOS logging pipeline and won't appear in the Inspector's 'Logs' tab. Instead, the logs are simply output to a file where you can inspect them. This means you can structure these logs however you like.

SpatialOS generates a raw worker log file for each worker instance when it starts a deployment. Worker instances can then write logs to this file for the duration of the deployment. However, for cloud deployments, the files are deleted when the deployment stops.

When debugging issues with worker instances, raw worker logs can be a useful tool for extracting debug information from the system. However, they lack much of the functionality of the normal SpatialOS logging infrastructure, so you should only use them in particular debugging scenarios.

For example:

  • When server-worker instances are having connection issues, you can obtain debug logs from raw worker logs.
  • When you want to extract a large amount of unstructured debug information from a worker instance, such as heap dumps, raw worker logs provide a convenient method for this.

Enabling raw worker logs

Raw worker logs are enabled by default in the GDKs for Unreal and Unity.

In order for the raw worker log file to be available to the worker instance, you need to add the ${IMPROBABLE_LOG_FILE} argument to the managed field of the worker configuration file. SpatialOS substitutes this argument with the location of the worker’s raw log file. You can then append logs to this file.

If you're using the flexible project layout (currently in beta), you should add ${IMPROBABLE_LOG_FILE} to the arguments field in your server-worker configuration file.

For more information, see Managed worker launch configuration.

Writing raw worker logs

Don't run out of space. Each node has about 30 GB disk space that is shared by multiple worker instances and their game assemblies.

In cloud deployments, stdout and stderr are written to <worker_id>_stdout_stderr.log, which is next to the ${IMPROBABLE_LOG_FILE} file.

Append to the ${IMPROBABLE_LOG_FILE} file as passed via your managed worker's launch configuration. You can create new files next to the ${IMPROBABLE_LOG_FILE} file for rotating logs.

Reading raw worker logs

Local deployments

The log files are located in the workspace at logs/<date>_<time>/workers/<worker>/<worker_id>.log.

Cloud deployments

There are two ways to access raw worker logs for cloud deployments:

  • From the Inspector

    In the Inspector sidebar, select a worker instance and then click Raw Worker Logs.

  • From the deployment page in the Console

    Select the Nodes tab followed by the worker_nanny node, and then click Raw logs.

Both of these methods take you to a simple web interface for inspecting the raw worker logs for the deployment. You can then find the raw logs for a particular worker instance at <worker>/<worker_id>.log.

Accessing crash dumps

When a server-worker instance crashes, it produces a crash dump, also known as a core dump. A crash dump is a file that contains a copy of the worker instance's memory when the crash occurs. You can use the crash dump to debug the crash.

Crash dumps of worker instances are automatically written to disk, and you can access them via the Raw logs interface from a running deployment.

To access a crash dump, complete the following steps:

  1. In the Console, open the running deployment, and click the Nodes tab.
  2. From the drop-down list, select the node that you want to view the crash dump of, and click Raw logs in the Node properties panel.
  3. Go to /data/improbable/logs/core-dumps/ and you can find the file for the crash dump. The following screenshot shows an example of a crash dump:

Note: The file name of the crash dump has the following pattern:



  • <worker_type> is the worker type of your worker instance.
  • <timestamp> is the time at which the crash occurred.
  • <worker_id> is the worker ID used to uniquely identify your worker instance.
  • <PID_inside_container> is the process ID of the worker inside the container.

2020-09-02 Page updated with editorial review: updated instruction and screenshot for reading raw worker logs
2019-08-14 Page updated with editorial review: added Accessing crash dumps

Updated about a year ago

Raw worker logs

Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.