Configuration for CLI is done using the YAML formatted configuration file which is placed by default in the root of the project and named as Marathonfile.

Below is an example of Marathonfile (without the vendor module configuration:

name: "sample-app tests"
outputDir: "./marathon"
analyticsConfiguration:
  influx:
    url: "http://influx.svc.cluster.local:8086"
    user: "root"
    password: "root"
    dbName: "marathon"
poolingStrategy:
- type: "omni"
shardingStrategy:
  type: "count"
  count: 5
sortingStrategy:
  type: "success-rate"
  timeLimit: "2015-03-14T09:26:53.590Z"
batchingStrategy:
  type: "fixed-size"
  size: 5
flakinessStrategy:
  type: "probability"
  minSuccessRate: 0.7
  maxCount: 3
  timeLimit: "2015-03-14T09:26:53.590Z"
retryStrategy:
  type: "fixed-quota"
  totalAllowedRetryQuota: 100
  retryPerTestQuota: 3

Each of these options is covered in detail in the section below. If you’re unsure how to properly format your options in Marathonfile take a look at the samples or take a look at the deserialisation logic in the cli module of the project. Each option might have a default deserialiser from yaml or a custom one. Usually the custom deserialiser expects the type option to understand which type of strategy we need to instantiate.

Configuration for gradle plugin is done via gradle only. It doesn’t support the CLI’s marathonfile. Here is an example of gradle config using Kotlin DSL:

marathon {
    name = "sample-app tests"
    baseOutputDir = "./marathon"
    analytics {
        influx {
            url = "http://influx.svc.cluster.local:8086"
            user = "root"
            password = "root"
            dbName = "marathon"
        }
    }
    poolingStrategy {
        operatingSystem = true
    }
    shardingStrategy {
        countSharding {
            count = 5
        }
    }
    sortingStrategy {
        executionTime {
            percentile = 90.0
            executionTime = Instant.now().minus(3, ChronoUnit.DAYS)
        }
    }
    batchingStrategy {
        fixedSize {
            size = 10
        }
    }
    flakinessStrategy {
        probabilityBased {
            minSuccessRate = 0.8
            maxCount = 3
            timeLimit = Instant.now().minus(30, ChronoUnit.DAYS)
        }
    }
    retryStrategy {
        fixedQuota {
            totalAllowedRetryQuota = 200
            retryPerTestQuota = 3
        }
    }
    filteringConfiguration {
        allowlist {
            add(SimpleClassnameFilter(".*".toRegex()))
        }
        blocklist {
            add(SimpleClassnameFilter("$^".toRegex()))
        }
    }
    testClassRegexes = listOf("^((?!Abstract).)*Test$")
    includeSerialRegexes = emptyList()
    excludeSerialRegexes = emptyList()
    uncompletedTestRetryQuota = 100
    ignoreFailures = false
    isCodeCoverageEnabled = false
    fallbackToScreenshots = false
    testOutputTimeoutMillis = 30_000
    strictMode = false
    debug = true
    autoGrantPermission = true
}

Here you will find a list of currently supported configuration parameters and examples of how to set them up. Keep in mind that some of the additional parameters might not be supported by all vendor modules. If you find that something doesn’t work - please submit an issue for a vendor module at fault.

General parameters
- Test run configuration name
- Output directory
Analytics configuration
Execution flow
Additional parameters
Vendor configuration

General parameters

Test run configuration name

This string specifies the name of this test run configuration. It is used mainly in the generated test reports.

```
name: "My test run for sample app"
```

marathon {
    name = "My test run for sample app"
}

marathon {
    name = "My test run for sample app"
}

Output directory

Directory path to use as the root folder for all the runner output (logs, reports, etc).

For gradle, the output path will automatically be set to a marathon folder in your reports folder unless it’s overridden.

```
outputDir: "build/reports/marathon"
```

marathon {
    baseOutputDir = "some-path"
}

marathon {
    baseOutputDir = "some-path"
}

Analytics configuration

Configuration of analytics backend to be used for storing and retrieving test metrics. This plays a major part in optimising performance and mitigating flakiness.

Disabled analytics

By default no analytics backend is expected which means that each test will be treated as a completely new test.

InfluxDB

Assuming you’ve done the setup for InfluxDB you need to provide:

url
username
password
database name
retention policy

Database name is quite useful in case you have multiple configurations of tests/devices and you don’t want metrics from one configuration to affect the other one, e.g. regular and end-to-end tests.

analyticsConfiguration:
  influx:
    url: "http://influx.svc.cluster.local:8086"
    user: "root"
    password: "root"
    dbName: "marathon"
    retentionPolicyConfiguration:
      name: "rpMarathonTest"
      duration: "90d"
      shardDuration: "1h"
      replicationFactor: 5
      isDefault: false

marathon {
    analytics {
        influx {
            url = "http://influx.svc.cluster.local:8086"
            user = "root"
            password = "root"
            dbName = "marathon"
        }
    }
}

marathon {
    analytics {
        influx {
            url = "http://influx.svc.cluster.local:8086"
            user = "root"
            password = "root"
            dbName = "marathon"
        }
    }
}

Graphite

Graphite can be used as an alternative to InfluxDB. It uses the following parameters:

host
port (optional) - the default is 2003
prefix (optional) - no metrics prefix will be used if not specified

analyticsConfiguration:
  graphite:
    host: "influx.svc.cluster.local"
    port: "8080"
    prefix: "prf"

marathon {
    analytics {
        graphite {
            host = "influx.svc.cluster.local"
            port = "8080"
            prefix = "prf"
        }
    }
}

marathon {
    analytics {
        graphite {
            host = "influx.svc.cluster.local"
            port = "8080"
            prefix = "prf"
        }
    }
}

Execution flow

Pooling strategy

Pooling strategy affects how devices are grouped together.

Omni a.k.a. one huge pool

All connected devices are merged into one group. This is the default mode.

```
poolingStrategy:
  - type: "omni"
```

marathon {
    //Omni is the default strategy
    poolingStrategy {}
}

marathon {
    //Omni is the default strategy
    poolingStrategy {}
}

By abi

Devices are grouped by their ABI, e.g. x86 and mips.

```
poolingStrategy:
  - type: "abi"
```

marathon {
    poolingStrategy {
        abi = true
    }
}

marathon {
    poolingStrategy {
        abi = true
    }
}

By manufacturer

Devices are grouped by manufacturer, e.g. Samsung and Yota.

poolingStrategy:
  - type: "manufacturer"

marathon {
    poolingStrategy {
        manufacturer = true
    }
}

marathon {
    poolingStrategy {
        manufacturer = true
    }
}

By device model

Devices are grouped by model name, e.g. LG-D855 and SM-N950F.

poolingStrategy:
  - type: "device-model"

marathon {
    poolingStrategy {
        model = true
    }
}

marathon {
    poolingStrategy {
        model = true
    }
}

By OS version

Devices are grouped by OS version, e.g. 24 and 25.

poolingStrategy:
  - type: "os-version"

marathon {
    poolingStrategy {
        operatingSystem = true
    }
}

marathon {
    poolingStrategy {
        operatingSystem = true
    }
}

Sharding strategy

Sharding is a mechanism that allows the marathon to affect the tests scheduled for execution inside each pool.

Parallel sharding

Executes each test in parallel on all the available devices in pool. This is the default behaviour.

```
shardingStrategy:
  type: "parallel"
```

marathon {
    //Parallel is the default strategy
    shardingStrategy {}
}

marathon {
    //Parallel is the default strategy
    shardingStrategy {}
}

Count sharding

Executes each test count times inside each pool. For example you want to test the flakiness of a specific test hence you need to execute this test a lot of times. Instead of running the build X times just use this sharding strategy and the test will be executed X times.

shardingStrategy:
  type: "count"
  count: 5

marathon {
    shardingStrategy {
        countSharding { 
            count = 5
        }
    }
}

marathon {
    shardingStrategy {
        countSharding { 
            count = 5
        }
    }
}

Sorting strategy

In order to optimise the performance of test execution tests need to be sorted. This requires analytics backend enabled since we need historical data in order to anticipate tests behaviour like duration and success/failure rate.

No sorting

No sorting of tests is done at all. This is the default behaviour.

```
sortingStrategy:
  type: "no-sorting"
```
```
marathon {
    sortingStrategy {}
}
```
```
marathon {
    sortingStrategy {}
}
```

Success rate sorting

For each test analytics storage is providing the success rate for a time window specified by time timeLimit parameter. All the tests are then sorted by the success rate in an increasing order, that is failing tests go first and successful tests go last. If you want to reverse the order set the ascending to true.

sortingStrategy:
  type: "success-rate"
  timeLimit: "2015-03-14T09:26:53.590Z"
  ascending: false

marathon {
    sortingStrategy {
        successRate { 
            limit = Instant.now().minus(Duration.parse("PT1H"))
            ascending = false
        }
    }
}

marathon {
    sortingStrategy {
        successRate { 
            limit = Instant.now().minus(Duration.parse("PT1H"))
            ascending = false
        }
    }
}

Execution time sorting

For each test analytics storage is providing the X percentile duration for a time window specified by time timeLimit parameter. Apart from absolute date/time it can be also be an ISO 8601 formatted duration.

Percentile is configurable via the percentile parameter.

All the tests are sorted so that long tests go first and short tests are executed last. This allows marathon to minimise the error of balancing the execution of tests at the end of execution.

sortingStrategy:
  type: "execution-time"
  percentile: 80.0
  timeLimit: "-PT1H"

marathon {
    sortingStrategy {
        executionTime { 
            percentile = 80.0
            timeLimit = Instant.now().minus(Duration.parse("PT1H"))
        }
    }
}

marathon {
    sortingStrategy {
        executionTime { 
            percentile = 80.0
            timeLimit = Instant.now().minus(Duration.parse("PT1H"))
        }
    }
}

Batching strategy

Batching mechanism allows you to trade off stability for performance. A group of tests executed using one single run is called a batch. Most of the times this means that between tests in the same batch you’re sharing the device state so there is no clean-up. On the other hand you gain some performance improvements since the execution command usually is quite slow (up to 10 seconds for some platforms).

Isolate batching

No batching is done at all, each test is executed using separate command execution, that is performance is sacrificed in favor of stability. This is the default mode.

```
batchingStrategy:
  type: "isolate"
```
```
marathon {
    batchingStrategy {}
}
```
```
marathon {
    batchingStrategy {}
}
```

Fixed size batching

Each batch is created based on the size parameter which is required. When a new batch of tests is needed the queue is dequeued for at most size tests.

Optionally if you want to limit the batch duration you have to specify the timeLimit for the test metrics time window and the durationMillis. For each test the analytics backend is accessed and percentile of it’s duration is queried. If the sum of durations is more than the durationMillis then no more tests are added to the batch.

This is useful if you have very very long tests and you use batching, e.g. you batch by size 10 and your test run duration is roughly 10 minutes, but you have tests that are expected to run 2 minutes each. If you batch all of them together then at least one device will be finishing it’s execution in 20 minutes while all other devices might already finish. To mitigate this just specify the time limit for the batch using durationMillis.

Another optional parameter for this strategy is the lastMileLength. At the end of execution batching tests actually hurts the performance so for the last tests it’s much better to execute them in parallel in separate batches. This works only if you execute on multiple devices. You can specify when this optimisation kicks in using the lastMileLength parameter, the last lastMileLength tests will use this optimisation.

batchingStrategy:
  type: "fixed-size"
  size: 5
  durationMillis: 100000
  percentile: 80.0
  timeLimit: "-PT1H"
  lastMileLength: 10

marathon {
    batchingStrategy {
        fixedSize { 
            size = 5
            durationMillis = 100000
            percentile = 80.0
            timeLimit = Instant.now().minus(Duration.parse("PT1H"))
            lastMileLength = 10
        }
    }
}

marathon {
    batchingStrategy {
        fixedSize { 
            size = 5
            durationMillis = 100000
            percentile = 80.0
            timeLimit = Instant.now().minus(Duration.parse("PT1H"))
            lastMileLength = 10
        }
    }
}

Flakiness strategy

This is the main anticipation logic for marathon. Using the analytics backend we can understand the success rate and hence queue preventive retries to mitigate the flakiness of the tests and environment.

Ignore flakiness

Nothing is done with this mode. This is the default behaviour.

```
flakinessStrategy:
  type: "ignore"
```
```
marathon {
    flakinessStrategy {}
}
```
```
marathon {
    flakinessStrategy {}
}
```

Probability based flakiness strategy

The main idea is that flakiness strategy anticipates the flakiness of the test based on the probability of test passing and tries to maximise the probability of passing when executed multiple times. For example the probability of test A passing is 0.5 and configuration has probability of 0.8 requested, then the flakiness strategy multiplies the test A to be executed 3 times (0.5 x 0.5 x 0.5 = 0.125 is the probability of all tests failing, so with probability 0.875 > 0.8 at least one of tests will pass).

The minimal probability that you want is specified using minSuccessRate during the time window controlled by the timeLimit. Additionally if you specify too high minSuccessRate you’ll have too many retries, so the upper bound for this is controlled by the maxCount parameter so that this strategy will calculate the required number of retries according to the minSuccessRate but if it’s higher than the maxCount it will choose maxCount.

flakinessStrategy:
  type: "probability"
  minSuccessRate: 0.7
  maxCount: 3
  timeLimit: "2015-03-14T09:26:53.590Z"

marathon {
    flakinessStrategy {
        probabilityBased { 
            minSuccessRate = 0.7
            maxCount = 3
            timeLimit = Instant.now().minus(Duration.parse("PT1H"))
        }
    }
}

marathon {
    flakinessStrategy {
        probabilityBased { 
            minSuccessRate = 0.7
            maxCount = 3
            timeLimit = Instant.now().minus(Duration.parse("PT1H"))
        }
    }
}

Retry strategy

This is the logic that kicks in if our preventive logic failed to anticipate such high number of retries. This works after the tests were actually executed.

No retries

As the name implies, no retries are done. This is the default mode.

```
retryStrategy:
  type: "no-retry"
```
```
marathon {
    retryStrategy {}
}
```
```
marathon {
    retryStrategy {}
}
```

Fixed quota retry strategy

Parameter totalAllowedRetryQuota specifies how many retries at all (for all the tests is total) are allowed. retryPerTestQuota controls how many retries can be done for each test individually.

retryStrategy:
  type: "fixed-quota"
  totalAllowedRetryQuota: 100
  retryPerTestQuota: 3

marathon {
    retryStrategy {
        fixedQuota { 
            retryPerTestQuota = 3
            totalAllowedRetryQuota = 100
        }
    }
}

marathon {
    retryStrategy {
        fixedQuota { 
            retryPerTestQuota = 3
            totalAllowedRetryQuota = 100
        }
    }
}

Additional parameters

Test filtering configuration

Filtering of tests is important since usually we as developers have the same codebase for all the different types of tests we want to execute. In order to indicate to marathon which tests you want to execute you can use the allowlist and blocklist parameters. First allowlist is applied, then the blocklist. Each accepts a TestFilter based on the class name, fully qualified class name, package, annotation or method. Each expects a regular expression as a value.

In order to filter using multiple filters at the same time a composition filter is also available which accepts a list of base filters and also an operation such as UNION, INTERSECTION or SUBTRACT. You can create complex filters such as get all the tests starting with E2E but get only methods from there ending with Test. Composition filter is not supported by groovy gradle scripts, but is supported if you use gradle kts.

An important thing to mention is that by default platform specific ignore options are not taken into account. This is because a cross-platform test runner cannot account for all the possible test frameworks out there. However, each framework’s ignore option can still be “explained” to marathon, e.g. JUnit’s org.junit.Ignore annotation can be specified in the filtering configuration.

filteringConfiguration:
  allowlist:
    - type: "simple-class-name"
      regex: ".*"
    - type: "fully-qualified-class-name"
      regex: ".*"
    - type: "method"
      regex: ".*"
    - type: "composition"
      filters:
        - type: "package"
          regex: ".*"
        - type: "method"
          regex: ".*"
      op: "UNION"
  blocklist:
    - type: "package"
      regex: ".*"
    - type: "annotation"
      regex: ".*"

marathon {
    filteringConfiguration {
        allowlist {
            simpleClassNameFilter = [".*"]
            fullyQualifiedClassnameFilter = [".*"]
            testMethodFilter = [".*"]
        }
        blocklist {
            testPackageFilter = [".*"]
            annotationFilter = [".*"]
        }
    }
}

marathon {
    filteringConfiguration { 
        allowlist = listOf(
            SimpleClassnameFilter(".*".toRegex()),
            FullyQualifiedClassnameFilter(".*".toRegex()),
            TestMethodFilter(".*".toRegex()),
            CompositionFilter(
                listOf(
                    TestPackageFilter(".*".toRegex()),
                    TestMethodFilter(".*".toRegex())
                ),
                CompositionFilter.OPERATION.UNION
            )
        )
        blocklist = listOf(
            TestPackageFilter(".*".toRegex()),
            AnnotationFilter(".*".toRegex())
        )
    }
}

Test class regular expression

By default, test classes are found using the "^((?!Abstract).)*Test[s]*$" regex. You can override this if you need to.

testClassRegexes:
  - "^((?!Abstract).)*Test[s]*$"

marathon {
    testClassRegexes = [
        "^((?!Abstract).)*Test[s]*\$"
    ]
}

marathon {
    testClassRegexes = listOf(
        "^((?!Abstract).)*Test[s]*$"
    )
}

Ignore failures

By default, the build fails if some tests failed. If you want to the build to succeed even if some tests failed use true.

```
ignoreFailures: true
```
```
marathon {
    ignoreFailures = true
}
```
```
marathon {
    ignoreFailures = true
}
```

Test output timeout

This parameter specifies the behaviour for the underlying test executor to timeout if there is no output. By default, this is set to 60 seconds.

```
testOutputTimeoutMillis: 30000
```

marathon {
    testOutputTimeoutMillis = 30000
}

marathon {
    testOutputTimeoutMillis = 30000
}

Test batch timeout

This parameter specifies the behaviour for the underlying test executor to timeout if the batch execution exceeded some duration. By default, this is set to 15 minutes.

```
testBatchTimeoutMillis: 900000
```

marathon {
    testBatchTimeoutMillis = 900000
}

marathon {
    testBatchTimeoutMillis = 900000
}

Device provider init timeout

When the test run starts device provider is expected to provide some devices. This should not take more than 3 minutes by default. If your setup requires this to be changed please override as following:

deviceInitializationTimeoutMillis: 300000

marathon {
    deviceInitializationTimeoutMillis = 300000
}

marathon {
    deviceInitializationTimeoutMillis = 300000
}

Analytics tracking

To better understand the use-cases that marathon is used for we’re asking you to provide us with anonymised information about your usage. By default, this is disabled. Use true to enable.

```
analyticsTracking: true
```

marathon {
    analyticsTracking = true
}

marathon {
    analyticsTracking = true
}

Uncompleted test retry quota

By default, tests that don’t have any status reported after execution (for example a device disconnected during the execution) retry indefinitely. You can limit the number of total execution for such cases using this option.

```
uncompletedTestRetryQuota: 100
```

marathon {
    uncompletedTestRetryQuota = 100
}

marathon {
    uncompletedTestRetryQuota = 100
}

Strict mode

By default, if one of the test retries succeeds then the test is considered successfully executed. If you require success status only when all retries were executed successfully you can enable the strict mode. This may be useful to verify that flakiness of tests was fixed for example.

```
strictMode: true
```
```
marathon {
    strictMode = true
}
```
```
marathon {
    strictMode = true
}
```

Debug mode

Enabled very verbose logging to stdout of all the marathon components. Very useful for debugging.

```
debug: true
```
```
marathon {
    debug = true
}
```
```
marathon {
    debug = true
}
```

Screen recording policy

By default, screen recording will only be pulled for tests that failed (ON_FAILURE option). This is to save space and also to reduce the test duration time since we’re not pulling additional files. If you need to save screen recording regardless of the test pass/failure please use the ON_ANY option:

```
screenRecordingPolicy: "ON_ANY"
```

marathon {
    screenRecordingPolicy = com.malinskiy.marathon.execution.policy.ScreenRecordingPolicy.ON_ANY
}

marathon {
    screenRecordingPolicy = com.malinskiy.marathon.execution.policy.ScreenRecordingPolicy.ON_ANY
}

Vendor configuration

See relevant vendor module page, e.g. Android or iOS