Preface


NOTE: This is the first edition of the project documentation, compiled in a form of a short book. It's still work in progress, which means a lot of errors and unfinished pages. If you've spotted some silly error, or want to add a paragraph somewhere, you can create an issue on github.

What does this book contain?

It contains a compilation of easy to read documents describing the project, it's various goals and elements.

It also covers things that are relevant to the process of creating simulation models and handling data using the proposed API.

Who is it intended for?

It's for anyone who wishes to understand the design behind outcome simulations, how they are run, how the data and simulation models (called collectively content) can be created and handled, among other things. Getting a good grip on the conceptual make up of the architecture is recommended before getting into creating content.

It's also for anyone who doesn't want to dive into specifics of how things are done "behind the scenes", but instead is interested in learning about creating content and learning how to use the provided tools. It's okay to skip the boring chapters and get into running simulations right away - one can always revisit certain chapters as you encounter problems later along the way.

How is it written - aiming for accessibility

Documentation for this project is meant to be easy to read and understand by anyone who doesn't have any prior experience with computer programming or game modding.

That said, this first edition is not kept up to this high standard of accessibility. It still contains things that can be considered to require some prior experience/knowledge related to the categories mentioned above. The goal for the next editions will be to further refine the documents in this book to:

  • reduce usage of highly "industry-specific" terms
  • increase the number and quality of explanations where the above is not possible
  • increase the number of links to external resources
  • provide illustrations to make things easier to understand

Introduction to the project

This page serves as a quick introduction to the project. It's written in a form of a list of answers to some of the frequently asked questions.

What is this project about?

It's about creation of user friendly environment for simulation model design and processing. It's about discovering possibilities for collaboration on certain kinds of simulation models.

At a more basic level it's about discovering a good minimal simulation architecture that's useful, extendable and easy to use.

What are the overall goals for this project?

  • provide a system for modeling and simulating social, economic as well as natural systems, and relationships between them
  • provide an inclusive environment for simulation modeling
  • provide a basic and easy to reason about simulation framework
  • provide a relatively easy to learn and simple to use interface for simulation modeling
  • provide a simple programmatic interface for interacting with models and simulations that can be used by custom applications

How useful is it right now?

Right now the project consists of a proposed system for how collaborative simulation-modeling could happen, as well as experimental software implementing things that are necessary for this to happen.

If you're ready to build from source (Rust programming language) you can already run some of the software.

See project overview for more information about the software sub-projects, and the project status page to learn more about what's being actively worked on right now.

How useful could it become?

That's hard to say. It depends on how useful the base simulation engine and it's API interface is.

It's designed to be relatively basic and generic so it can scale well, but it's not certain that it will.

The design of the engine itself imposes important limitations on the possible simulations to be created for it. There are trade-offs to be had, as with most things, and the overall design here is influenced by the larger goals of the project.

Community created content?

The goal is to create a situation where multiple users can collaborate on files organized into versioned modules.

User files (for the sake of simplicity also collectively called content) are parsed and a simulation instance is spawned using that data.

User files provide both the initial state information (here state meaning a data-based representation of an object at some point in simulation time; we call this data) as well as the computation instructions necessary for running the simulation.

Project Overview

This project is not monolithic, it's composed of a bunch of smaller subprojects.

As mentioned in the introduction, at the core of the project lies the simulation engine itself. It handles functionality related to parsing and running simulations, exposing a simple interface for interacting with the simulation data to the programmer.

Then there are the tools that provide an environment for working with the simulations. Tools also provide new layers of functionality and interoperability, for example offering a network interface (see endgame).

Finally there are games and other applications.

Simulation engine

The engine is not an executable application. It's a library that can be used by other applications to create and run simulations.

Simulation engine handles:

  • parsing input data
  • creating simulation instances
  • processing simulation instances
  • reading and writing simulation instance data

Implementation details aside, these tasks enable applications to make use of outcome simulations, including .

The engine takes care of all the details of creating and processing simulations, using the library doesn't require complete knowledge of how it works. To learn about how the engine works check out the next chapter.

Tools

To be able to run simulations we need some kind of application that makes use of the simulation engine.

The most basic tool is the command-line based endgame. Some of it's functionality is also exported as a library so it can be used within other applications. One useful example is the networking layer functionality.

Command-line is not for everyone, that's why there's also furnace, which is a GUI app with a window-based interface. It works on all popular operating systems (Linux, Windows, Mac). furnace is not exactly a replacement or alternative for endgame, rather it builds on it's features to be even more useful to the user.

Games and other applications

One good example of incorporating outcome simulations in different kinds of projects are games.

Anthropocene is a modern-day global strategy game. It doesn't use the simulation engine library directly, instead it uses the networking layer provided by endgame. It serves as a demonstration of how all kinds of projects, including games, can make use of outcome simulations no matter the framework or the programming language used - the only requirement here is the ability to send and receive data using a tcp connection.

Basic concepts

In this chapter we’ll take a high-level perspective on some of the basic concepts behind the simulations.

It's recommended to go through this chapter to get an understanding of the basic ideas around which the system is organized.

If you're not so much interested in learning all that now, and want to get to creating mods and scenarios right away instead, check out the light-weight guided intro to modding under the guides and tutorials section.

At a glance

  • simulation models and data are created by users, therefore
  • the engine itself doesn't contain simulation models or data ("moving parts" like entities and components are generic, almost nothing is hardcoded)
  • data-driven architecture, everything is based on addressed variables (including "event-like" behavior)
  • processing scheme based on single clock, multiple clock events
  • logic based on state machines, each state containing a set of executable commands ("micro programs")
  • built-in separation of the entity objects and their data from each other, reflected in the way inter-entity data operations are written compared to intra-entity

endgame 0.1.3

endgame is one of the tools in development right now. It's a command-line interface (cli) tool, meaning it involves interacting with the command-line interface (shell) provided by your operating system.

If you're not sure how to access the shell on your operating system consult an appropriate tutorial for that.

Running the program

You can either download and run a pre-compiled binary or compile the program yourself.

Downloading binary

Probably the easiest way to run endgame is to download a binary suitable for your system and run it. You'll find compiled binaries attached to the bigger releases. Note that the binaries may not be available for every release, also not all architectures/operating systems are covered.

To run the program you will need to navigate to the directory where you saved the binary from the command line. Then type in the name of the program to run it.

If you don't know how to run a binary file from the command line please consult the nearest web search.

Compiling yourself

You can also compile endgame yourself.

To do this you will need to install the Rust programming language on your machine. Installing Rust is easy, simply follow the instructions on rustup.rs. Installing git is also a good idea, but is not required.

Clone the repository with git

git clone https://github.com/adamsky/endgame

Alternatively just download the repository as zip archive and unpack it.

From the command line, change the directory to wherever you cloned/unpacked the repository. Then run

cargo run --release

Cargo is the package manager for Rust programming language. It takes care of downloading all dependencies, building everything, and finally running the program. The --release flag turns on optimizations and makes our program faster in the end, but it makes the build process itself longer.

In this guide we'll be passing arguments to the endgame program. You can pass those arguments right from the cargo run command.

cargo run --release -- --help

Anything after the -- will be passed to our program.

Available functionality

Running endgame you should be greeted with something similar to the following:

endgame 0.1.2
Adam Adamsky <adamadamsky@protonmail.com>
Endgame is a command line toolkit for creating,
running and analyzing outcome simulations.

It's is part of the effort to create an accessible
playground for world modeling and simulation.
For more info check out https://theoutcomeproject.com

USAGE:
    endgame [FLAGS] <SUBCOMMAND>

FLAGS:
    -d               Print debug information verbosely
    -h, --help       Prints help information
    -V, --version    Prints version information

SUBCOMMANDS:
    init      Initialize new content data structure
    lint      Check content for errors
    test      Test content configuration for memory requirements, average processing speed, etc.
    run       Run simulation
    server    Run endgame in server mode.
    client    Run endgame in client mode.
    coord     Run a cluster coordinator.
    worker    Run a cluster worker.
    gen       Generate optimized binary.
    help      Prints this message or the help of the given subcommand(s)

Descriptions for each subcommand tell us what it is they do. For additional information we can run any of the subcommands with an added --help flag to learn more.

For this guide we will not be looking into all of the subcommands. We will focus specifically on run.


NOTE: endgame is currently in an early version, a lot of the features are not yet useful or are simply placeholders. The functionality and it's specific layout (API) is likely to change over the course of development.

Interactive runner

Most interesting feature available to us in the current version is the interactive simulation runner under the run subcommand.

./endgame run <path-to-scenario>

--interactive option is by default set to true, so this will start an interactive session.

Let's run the scenario we made in the guide called "first scenario".

You should be greeted with a few lines looking similar to this:

Running interactive session using scenario at: ".../endgame/test_scenario"
[INFO] generating sim instance from scenario at: .../endgame/test_scenario/
[INFO] there are 1 mods listed in the scenario manifest
[INFO] mod found: "test_module" version: "0.1.0" (version specifier in scenario manifest: "^0.1.0")
[INFO] found all mods listed in the scenario manifest (1)
[INFO] successfully created sim_instance
You're now in interactive mode.
See possible commands with "help". Exit using "quit" or ctrl-d.

Config file interactive.yaml doesn't exist, loading default config settings
[1]

The interactive mode enables you to interact with the simulation as it's being run. There are a few different commands that enable you to do a bunch of different things. Run help to see all available commands. You can use TAB key to navigate through the commands more easily too (autocomplete). Try just typing "h" and then press TAB twice, it should show you possible commands beginning with "h".

Simply pressing enter with no input ("empty command") progresses the simulation by one turn. A turn is a number of base simulation ticks.

cfg and show

You can set the number of ticks per turn using the configuration (cfg) variable ticks_per_turn. By default it's set to 1. Let's change this and make one turn do 24 ticks:

cfg ticks_per_turn 24

Preview the list of currently set cfg variables with:

cfg-list
ticks_per_turn          24
show_on                 true
show_list               []

Now when we process one turn the prompt should change by 24 at a time.

show command shows takes data from the simulation and shows it to us. It uses the show_list cfg variable as input - it's a list of addresses that can be used to pull data from the simulation. Let's add an address to the show_list.

show-add /region/greenland/property/bear_population/int/count

Now when you use the show command it should print out the value from the address.

Of course the address has to point to a variable that exists in the simulation.

show_on config variable defines whether on each turn show should be invoked. We can easily toggle this with show-toggle.

You can export the current configuration to file with

cfg-save

Adding and modifying the variables from the command line can be tiresome, you can edit the cfg file itself (interactive.yaml) and then just use

cfg-reload

to load it into the currently running interactive session. Every time an interactive session is started it will look for this file in the current working directory and automatically load it if it exists.

run and run-until

run command takes in a number of base hour ticks and executes exactly that number. run-until takes in an integer and will run the simulation until the simulation clock count is equal to that number. You can use CTRL-D (EOF) or CTRL-C to break out of run and run-until. There are also runf and runf-until which are faster but don't really allow for breaking out of the execution (because they're not taking any time to listen for signals while running).

Let's make a clock

endgame 0.1.3, outcome 0.2.2

https://github.com/theoutcome/clock_tutorial

Building on what you've learned so far, let's create a clock.

Our clock will consist of hours, days, months and years. Once we build an appropriate module for the clock, we will also configure endgame to display our clock in the prompt of the interactive runner, like so:

You're now in interactive mode.
See possible commands with "help". Exit using "quit" or ctrl-d.

Loading config settings from file (found interactive.yaml)
[1-1-2015 1:00]
[1-1-2015 2:00]
[1-1-2015 3:00]
[1-1-2015 4:00]
[1-1-2015 5:00]
(...)

Clock module

Let's create a new scenario called clock_tutorial.

HINT: You can use endgame to initialize a scenario for you if you don't want to create the files manually.

WIP

endgame configuration

We can change what endgame displays in the interactive runner's prompt. To do this we will need to include the following in the interactive.yaml config file:

prompt_format: "{}-{}-{} {}:00"
prompt_vars:
- /uni/uni/generic/clock/str/day
- /uni/uni/generic/clock/str/month
- /uni/uni/generic/clock/str/year
- /uni/uni/generic/clock/str/hour

REMINDER: endgame by default loads configuration from a file called interactive.yaml from inside the directory where it's run from. If the file doesn't exist you can just create it. Alternatively use cfg-save while in the interactive mode to export current config to file.

Now you can either quit and start the interactive runner again, or just use the cfg-reload command. The second option is useful because you don't have to exit the current simulation run in order to change the prompt. Instead you can tweak the config and reload it inside the interactive runner at any time.

Note that if any of the addresses listed in prompt_vars is not reachable then the prompt will ignore the custom format entirely and default to showing the usual tick count number.

Creating a modifier

WIP

Introduction to Lua scripting

outcome 0.2.2

WIP

Regular commands include most basic operations we would want to do on our data. Simple calculations, evaluations, getting and setting variables can all be done without using any complex scripts.

Whenever we have the need to include more complex algorithms however, we have the ability to include Lua scripts in our models.

To make use of Lua scripts you will need to master at least the basics of Lua. If you didn't have any contact with programming up until now, don't be discouraged. You will be able to follow this tutorial just fine.

What is Lua

Lua is a lightweight programming language designed primarily for embedded use in applications, one of the most widely used scripting languages.

If you don't know much about Lua there are lots of resources to learn from online.

Performance question

Lua itself is highly optimized and really quite fast. But the Lua scripts we include in our components will always be slower than the bare commands. This is something to keep in mind while designing your model.

Ideally we will only want to use Lua for complex tasks that occur infrequently during the processing. Having a complex Lua script execute many times every simulation tick can have a noticeable effect on performance.

Declaring a simple lua_script

Here's a map representation of a simple lua_script cmd.

- cmd: lua_script
  src: |
    print("Hello world!")

inputs and outputs

The scripts we pass to our lua_script command don't have access to any of the variables we can normally access. Instead we're required to specify what data they will get and what data we will get out from them after they're finished executing. It really is a quite simple arrangement.

Inputs are passed to the script as it's global variables. Outputs are

- cmd: lua_script
  inputs:
    # lua global named "string_var" will be set to the value of `str/local_string`
    string_var: str/local_string
  outputs:
    string_var: str/string_var
  src: |
    print("Hello world!")