The prompt and accurate execution of liquidations and disputes is a core assumption to all priceless financial contracts compatible with the UMA DVM. Liquidation and dispute bots, as described below and implemented here and here, are infrastructure tools that will help maintain the overall health of the UMA ecosystem. They are currently compatible with the priceless synthetic token contract template, as described here and implemented here.
Liquidation vs Dispute Bot
The liquidation bot monitors all open positions within a given expiring multi-party contract and liquidates positions if their collateralization ratio, as inferred from off-chain information about the value of the price identifier, drops below a given threshold.
The dispute bot monitors all liquidations occurring within a given expiring multi-party contract and initiates disputes against liquidations it deems invalid, as inferred from off-chain information about the value of the price identifier. A liquidation is invalid if a position was correctly collateralized at the time of liquidation.
In short, a liquidation bot is to liquidate under-collateralized positions while a dispute bot is used for disputing those liquidations if they incorrectly report a position as under-collateralized. Since a dispute is the only way a price request is triggered in the system, they are very important to the operation of the DVM.
Incentives to Running a Bot
Details about liquidation and dispute rewards can be found here.
The liquidation and dispute bots are separate entities. Each has its own wallet and is designed to be be run independently of the other. This decouples dependencies between the bots to decrease the risk of one impacting the other. In production, it is suggested to run the bots within docker containers to isolate the bots and to ensure a reproducible execution environment.
This tutorial will be broken down into three main sections:
- Running the bots directly from your host environment (no Docker) from the command line
- Running the bots within a dockerized environment from the official UMA Docker image
- Deploying bots to production in Google Cloud Compute
This tutorial will guide you through setting up a liquidator and disputer to monitor an expiring multi party deployed on the Kovan test network. A verified version of the expiring multi party contract can be found on Kovan here. This contract is an ETHBTC synthetic, collateralized in Dai.
Before starting this tutorial you need to clone the repo, install the dependencies, and compile the smart contracts.
If you are using Linux, you may need to first install the following packages before
yarn can complete properly:
libusb-1.0-0 may have different names depending on your Linux distribution.
If you are not using Linux (or if you have installed the above), you can run the following to continue:
If you have any issues executing
npx truffle <command> you can try running
$(npm bin)/truffle <command>. If you have Truffle globally installed you should be able to run
truffle <command> without npx or
$(npm bin) but it may not be the same version specified in the project.
Bots must be funded with currency to pay for liquidations and disputes. Specifically, if you want to run a liquidation bot, you need to fund the wallet with 1) synthetic tokens to close liquidated positions, 2) collateral currency to pay the DVM final fee, and 3) Ether to pay transaction fees. If you want to run a dispute bot, then this wallet should be funded with 1) collateral currency to pay dispute bonds and the DVM final fee and 2) Ether to pay for transactions.
At a minimum, the account must have some ETH to pay gas for transactions executed during bot set up, otherwise the bots will crash. The Kovan ETH faucet can be found here. If the bots do not have collateral or synthetic tokens, they will not crash but they will not be able to send liquidations or disputes.
Bot private key management
All deployment configurations require a wallet mnemonic (or private key) to be injected into the bots enabling them to submit transactions to perform on-chain liquidations and disputes.
You can either bring your own mnemonic from an existing wallet or generate a fresh one using the
bip39 package installed within the UMA repo.
If you have a wallet mnemonic already you can skip this section.
To generate a new mnemonic you can run the following from the
You can then load this mnemonic into truffle and view the associated address. To do this, set the mnemonic as an environment variable by running:
You can now fund this wallet with the associated currency for the type of bot you want to run. To learn more about creating synthetic tokens to fund your liquidation bot see this tutorial.
Running the liquidator and disputer bots locally
This tutorial start with running the liquidator and disputer bots locally from your machine without docker or any complex execution environment. This is meant to be the simplest way posable to start up a bot.
a) Configuring environment
To start a bot the first step is to configure the bot's settings.
Liquidation bots require 4 main configurations settings which are configured using environment variables.
To set this up create a
.env file in the
/core directory of the repo it:
The parameters above, as well as other optional parameters are explained in the appendix of this tutorial. Be sure to add in your mnemonic and your crypto watch API key. The parameter in the example above conform to UMIP-2's specification.
Note that the
EMP_ADDRESS above currently refers to the ETHBTC synthetic token deployed on Kovan. The EMP refers to the
ExpiringMultiParty financial contract used here.
b) Starting the bots
Now that your env is set up you can run the bot. Run the following command from the
core directory to start the bots on Kovan:
This will start the liquidator bot process using the network
kovan and the wallet
mnemonic. You should see the following output:
In a separate terminal you can start a disputer bot using the same config by running:
You should see the following output:
The bots are now running! Any liquidation event or informative logs will be printed here. If all is operating correctly, you should be able to liquidate under-collateralized positions and dispute incorrectly liquidated positions.
Running the bots locally with Docker
Running the bots from your local machine is simple but is not ideal for long term bot execution as you'll need to keep a terminal running the whole time to keep the bot process alive. What would be better is to run the bots within an isolated Docker container that can be run in the background on your machine.
The steps followed here can also be reproduced on a VPS service like Google Cloud Compute Engine, Digital Ocean or AWS EC2 to host your Dockerized bots in the cloud. Section 3 of this tutorial will show you how to deploy the bots to GCP. It is recommended that you get the bots running within a local Docker environment before trying to run them on the cloud.
a) Setting up docker on your local machine
To be able to run the bots within a Dockerized environment you need to have Docker set up on your local machine. Getting started with Docker is relatively straight forward. See the official docs for Mac, Linux or Windows.
b) Creating environment configs for the bots
In the previous section, both the liquidator and disputer bots used the same
.env configuration file.
In this section, we will create separate configuration files for each bot.
These scripts will contain all the settings for a given bot, as well as the starting command used to boot the bot.
Start by copying the
.env you created to make two new env files. This section assumes you are in the
/core directory. Run the following commands:
You should now have two config files
disputer.env within the
/core directory which contain the original configs defined in the previous section along with a
COMMAND line which defines the execution command of the bot.
These commands are the same as before.
c) Starting the Docker containers
Next, we will start the Docker containers in detached mode on our local machine. To do this run the following:
Both docker containers should start correctly. To view the running docker containers on your machine you can run:
You can now view the logs of these docker containers by running:
This should print the logs of the bot's execution to date.
You can also run in attached mode to view the logs as they are printed as:
If you want to stop the bots from running you can run the following:
Note this will stop all running docker containers on your machine.
Running the bots in the cloud with GCP
In this section of the tutorial you will learn how to spin up a liquidator and dispute bot in Google Cloud Compute. Note that you can use any cloud hosting platform to run the Docker containers. We simply use GCP as an easy example. The official GCP docs are a useful reference. For more information and commands see this official tutorial.
a) Setting up your
Before running any GCP commands you need to set up an account and configure the
See this official documentation on setting this up. Once this is set up you should be able to run the following to view your gcloud email:
b) Deploying Bots to GCP
To deploy the bots to GCP we use the
compute instances create-with-container CLI function which will create a new compute instance within your GCP Compute engine.
This instance will boot up and run a Docker container on execution.
From the core directory, where
disputer.env configs are located, you can run the following to deploy bots to GCP:
ethbtc-liquidator-kovan is the name of the deployed compute instance. You can change this to anything you want.
./liquidator.env is the environment configuration file created in the previous section of this tutorial.
Logging into gcloud Compute Engine dashboard you should see a deployed compute instance called
You can run the same command with a new name and new configuration file to start the disputer bot.
C) Updating container configuration.
You might want to update docker environment variables or push new code to the compute instances. The easiest way to update env variables is from the GCP dashboard. The official docs explain the process to do this here.
Alternatively if you want to push a new local config file you can do this by running the following command (assuming we are updating the
ethbtc-liquidator-kovan initialized in the previous code block)
Bot health monitoring
Once your bots are running on GCP you want to know when they are properly executing liquidations or if any errors occur. The bots by default provide a few useful monitoring avenues. These are now discussed.
GCP StackDriver integration
Once your containers are running within compute engine you probably want to see the log messages in the same way you did when running
docker attach in the previous section.
To make this process easier the bot logger configuration includes Google Stackdriver integration.
This enables GCP logging to pick up the log messages generated from the bots. To enable this add the following environment variable to the docker containers:
This will tell the bot's logger to pipe logs to GCP logging. For more information on this GCP logging and how to access it within Gcloud see the official doc here.
The bots by default also come with built in slack integration to send slack messages when events occur (bot liquidates a position, is running low on capital or something has failed). To use this integration you must first generate a Slack webhook. Information on generating a webhook can be found on the slack docs here.
Once you've set up a Slack webhook you can add it to the bots by adding the following environment variable:
The bots will now automatically start sending log messages in Slack.
Running a liquidator bot on mainnet
The tutorial thus far assumed you are running Kovan liquidation and dispute bots. Next, we will discuss how to move the deployment onto Ethereum mainnet. This involves three main steps which are outlined below.
1) Funding your liquidator bot's mainnet wallet
Running on mainnet involves first repeating the funding accounts section on the main Ethereum network to acquire collateral to fund the liquidator and disputer bots.
2) Updating the EMP address to point to mainnet expiring multi party contract
Update your environment configuration
EMP_ADDRESS to refer to the mainnet address of the expiring multiparty contract you want to monitor.
3) Update the
COMMAND used to start the bots to point at mainnet, rather than kovan.
This is as simple as changing your
COMMAND to the following for the liquidator and disputer bots respectively.
Specifying liquidation sensitivity parameters
Due to the design of the UMA DVM and liquidation process, a bot operator might only want to liquidate positions that are
x percentage under-collateralized. For example if the Collateralization Requirement (CR) of a contract is 120%, a conservative operator might only want to liquidate positions that are at 115% to remove any risk of being disputed.
To facilitate this configuration, the UMA liquidator bots have an optional configuration object that can be used to specify a
crThreshold which defines how far below the given CR ratio a position must fall before the bot will initiate the liquidation. To enable this configuration with a 5% threshold, add the following to your
This configuration object has additional options including:
liquidationDeadline(in seconds) which aborts the liquidation if the transaction is mined this amount of time after the EMP client's last update time. Defaults to 5 minutes.
liquidationMinPrice(in Wei, as a
FixedPointtype) which aborts the liquidation if the amount of collateral in the position per token outstanding is below this ratio. Defaults to 0.
These configurations can be added to the config in the same way the
crThreshold was added (i.e. as part of the JSON object in the same line).
Bot configuration parameters
For a more detailed document on bot-specific parameters see this doc.