Configuration for CLI is done using the YAML formatted configuration file which is placed by default in the root of the project and named as Marathonfile.
Below is an example of Marathonfile (without the vendor module configuration:
name: "sample-app tests"
outputDir: "./marathon"
analyticsConfiguration:
influx:
url: "http://influx.svc.cluster.local:8086"
user: "root"
password: "root"
dbName: "marathon"
poolingStrategy:
- type: "omni"
shardingStrategy:
type: "count"
count: 5
sortingStrategy:
type: "success-rate"
timeLimit: "2015-03-14T09:26:53.590Z"
batchingStrategy:
type: "fixed-size"
size: 5
flakinessStrategy:
type: "probability"
minSuccessRate: 0.7
maxCount: 3
timeLimit: "2015-03-14T09:26:53.590Z"
retryStrategy:
type: "fixed-quota"
totalAllowedRetryQuota: 100
retryPerTestQuota: 3
Each of these options is covered in detail in the section below. If you’re unsure how to properly format your options in Marathonfile take a look at the samples or take a look at the deserialisation logic in the cli module of the project. Each option might have a default deserialiser from yaml or a custom one. Usually the custom deserialiser expects the type option to understand which type of strategy we need to instantiate.
Configuration for gradle plugin is done via gradle only. It doesn’t support the CLI’s marathonfile. Here is an example of gradle config using Kotlin DSL:
marathon {
name = "sample-app tests"
baseOutputDir = "./marathon"
analytics {
influx {
url = "http://influx.svc.cluster.local:8086"
user = "root"
password = "root"
dbName = "marathon"
}
}
poolingStrategy {
operatingSystem = true
}
shardingStrategy {
countSharding {
count = 5
}
}
sortingStrategy {
executionTime {
percentile = 90.0
executionTime = Instant.now().minus(3, ChronoUnit.DAYS)
}
}
batchingStrategy {
fixedSize {
size = 10
}
}
flakinessStrategy {
probabilityBased {
minSuccessRate = 0.8
maxCount = 3
timeLimit = Instant.now().minus(30, ChronoUnit.DAYS)
}
}
retryStrategy {
fixedQuota {
totalAllowedRetryQuota = 200
retryPerTestQuota = 3
}
}
filteringConfiguration {
allowlist {
add(SimpleClassnameFilter(".*".toRegex()))
}
blocklist {
add(SimpleClassnameFilter("$^".toRegex()))
}
}
testClassRegexes = listOf("^((?!Abstract).)*Test$")
includeSerialRegexes = emptyList()
excludeSerialRegexes = emptyList()
uncompletedTestRetryQuota = 100
ignoreFailures = false
isCodeCoverageEnabled = false
fallbackToScreenshots = false
testOutputTimeoutMillis = 30_000
strictMode = false
debug = true
autoGrantPermission = true
}
Here you will find a list of currently supported configuration parameters and examples of how to set them up. Keep in mind that some of the additional parameters might not be supported by all vendor modules. If you find that something doesn’t work - please submit an issue for a vendor module at fault.
This string specifies the name of this test run configuration. It is used mainly in the generated test reports.
name: "My test run for sample app"
marathon {
name = "My test run for sample app"
}
marathon {
name = "My test run for sample app"
}
Directory path to use as the root folder for all the runner output (logs, reports, etc).
For gradle, the output path will automatically be set to a marathon
folder in your reports folder unless it’s overridden.
outputDir: "build/reports/marathon"
marathon {
baseOutputDir = "some-path"
}
marathon {
baseOutputDir = "some-path"
}
Configuration of analytics backend to be used for storing and retrieving test metrics. This plays a major part in optimising performance and mitigating flakiness.
By default no analytics backend is expected which means that each test will be treated as a completely new test.
Assuming you’ve done the setup for InfluxDB you need to provide:
Database name is quite useful in case you have multiple configurations of tests/devices and you don’t want metrics from one configuration to affect the other one, e.g. regular and end-to-end tests.
analyticsConfiguration:
influx:
url: "http://influx.svc.cluster.local:8086"
user: "root"
password: "root"
dbName: "marathon"
retentionPolicyConfiguration:
name: "rpMarathonTest"
duration: "90d"
shardDuration: "1h"
replicationFactor: 5
isDefault: false
marathon {
analytics {
influx {
url = "http://influx.svc.cluster.local:8086"
user = "root"
password = "root"
dbName = "marathon"
}
}
}
marathon {
analytics {
influx {
url = "http://influx.svc.cluster.local:8086"
user = "root"
password = "root"
dbName = "marathon"
}
}
}
Graphite can be used as an alternative to InfluxDB. It uses the following parameters:
analyticsConfiguration:
graphite:
host: "influx.svc.cluster.local"
port: "8080"
prefix: "prf"
marathon {
analytics {
graphite {
host = "influx.svc.cluster.local"
port = "8080"
prefix = "prf"
}
}
}
marathon {
analytics {
graphite {
host = "influx.svc.cluster.local"
port = "8080"
prefix = "prf"
}
}
}
Pooling strategy affects how devices are grouped together.
All connected devices are merged into one group. This is the default mode.
poolingStrategy:
- type: "omni"
marathon {
//Omni is the default strategy
poolingStrategy {}
}
marathon {
//Omni is the default strategy
poolingStrategy {}
}
Devices are grouped by their ABI, e.g. x86 and mips.
poolingStrategy:
- type: "abi"
marathon {
poolingStrategy {
abi = true
}
}
marathon {
poolingStrategy {
abi = true
}
}
Devices are grouped by manufacturer, e.g. Samsung and Yota.
poolingStrategy:
- type: "manufacturer"
marathon {
poolingStrategy {
manufacturer = true
}
}
marathon {
poolingStrategy {
manufacturer = true
}
}
Devices are grouped by model name, e.g. LG-D855 and SM-N950F.
poolingStrategy:
- type: "device-model"
marathon {
poolingStrategy {
model = true
}
}
marathon {
poolingStrategy {
model = true
}
}
Devices are grouped by OS version, e.g. 24 and 25.
poolingStrategy:
- type: "os-version"
marathon {
poolingStrategy {
operatingSystem = true
}
}
marathon {
poolingStrategy {
operatingSystem = true
}
}
Sharding is a mechanism that allows the marathon to affect the tests scheduled for execution inside each pool.
Executes each test in parallel on all the available devices in pool. This is the default behaviour.
shardingStrategy:
type: "parallel"
marathon {
//Parallel is the default strategy
shardingStrategy {}
}
marathon {
//Parallel is the default strategy
shardingStrategy {}
}
Executes each test count times inside each pool. For example you want to test the flakiness of a specific test hence you need to execute this test a lot of times. Instead of running the build X times just use this sharding strategy and the test will be executed X times.
shardingStrategy:
type: "count"
count: 5
marathon {
shardingStrategy {
countSharding {
count = 5
}
}
}
marathon {
shardingStrategy {
countSharding {
count = 5
}
}
}
In order to optimise the performance of test execution tests need to be sorted. This requires analytics backend enabled since we need historical data in order to anticipate tests behaviour like duration and success/failure rate.
No sorting of tests is done at all. This is the default behaviour.
sortingStrategy:
type: "no-sorting"
marathon {
sortingStrategy {}
}
marathon {
sortingStrategy {}
}
For each test analytics storage is providing the success rate for a time window specified by time timeLimit parameter.
All the tests are then sorted by the success rate in an increasing order, that is failing tests go first and successful tests go last.
If you want to reverse the order set the ascending
to true
.
sortingStrategy:
type: "success-rate"
timeLimit: "2015-03-14T09:26:53.590Z"
ascending: false
marathon {
sortingStrategy {
successRate {
limit = Instant.now().minus(Duration.parse("PT1H"))
ascending = false
}
}
}
marathon {
sortingStrategy {
successRate {
limit = Instant.now().minus(Duration.parse("PT1H"))
ascending = false
}
}
}
For each test analytics storage is providing the X percentile duration for a time window specified by time timeLimit parameter. Apart from absolute date/time it can be also be an ISO 8601 formatted duration.
Percentile is configurable via the percentile parameter.
All the tests are sorted so that long tests go first and short tests are executed last. This allows marathon to minimise the error of balancing the execution of tests at the end of execution.
sortingStrategy:
type: "execution-time"
percentile: 80.0
timeLimit: "-PT1H"
marathon {
sortingStrategy {
executionTime {
percentile = 80.0
timeLimit = Instant.now().minus(Duration.parse("PT1H"))
}
}
}
marathon {
sortingStrategy {
executionTime {
percentile = 80.0
timeLimit = Instant.now().minus(Duration.parse("PT1H"))
}
}
}
Batching mechanism allows you to trade off stability for performance. A group of tests executed using one single run is called a batch. Most of the times this means that between tests in the same batch you’re sharing the device state so there is no clean-up. On the other hand you gain some performance improvements since the execution command usually is quite slow (up to 10 seconds for some platforms).
No batching is done at all, each test is executed using separate command execution, that is performance is sacrificed in favor of stability. This is the default mode.
batchingStrategy:
type: "isolate"
marathon {
batchingStrategy {}
}
marathon {
batchingStrategy {}
}
Each batch is created based on the size parameter which is required. When a new batch of tests is needed the queue is dequeued for at most size tests.
Optionally if you want to limit the batch duration you have to specify the timeLimit for the test metrics time window and the durationMillis. For each test the analytics backend is accessed and percentile of it’s duration is queried. If the sum of durations is more than the durationMillis then no more tests are added to the batch.
This is useful if you have very very long tests and you use batching, e.g. you batch by size 10 and your test run duration is roughly 10 minutes, but you have tests that are expected to run 2 minutes each. If you batch all of them together then at least one device will be finishing it’s execution in 20 minutes while all other devices might already finish. To mitigate this just specify the time limit for the batch using durationMillis.
Another optional parameter for this strategy is the lastMileLength. At the end of execution batching tests actually hurts the performance so for the last tests it’s much better to execute them in parallel in separate batches. This works only if you execute on multiple devices. You can specify when this optimisation kicks in using the lastMileLength parameter, the last lastMileLength tests will use this optimisation.
batchingStrategy:
type: "fixed-size"
size: 5
durationMillis: 100000
percentile: 80.0
timeLimit: "-PT1H"
lastMileLength: 10
marathon {
batchingStrategy {
fixedSize {
size = 5
durationMillis = 100000
percentile = 80.0
timeLimit = Instant.now().minus(Duration.parse("PT1H"))
lastMileLength = 10
}
}
}
marathon {
batchingStrategy {
fixedSize {
size = 5
durationMillis = 100000
percentile = 80.0
timeLimit = Instant.now().minus(Duration.parse("PT1H"))
lastMileLength = 10
}
}
}
This is the main anticipation logic for marathon. Using the analytics backend we can understand the success rate and hence queue preventive retries to mitigate the flakiness of the tests and environment.
Nothing is done with this mode. This is the default behaviour.
flakinessStrategy:
type: "ignore"
marathon {
flakinessStrategy {}
}
marathon {
flakinessStrategy {}
}
The main idea is that flakiness strategy anticipates the flakiness of the test based on the probability of test passing and tries to maximise the probability of passing when executed multiple times. For example the probability of test A passing is 0.5 and configuration has probability of 0.8 requested, then the flakiness strategy multiplies the test A to be executed 3 times (0.5 x 0.5 x 0.5 = 0.125 is the probability of all tests failing, so with probability 0.875 > 0.8 at least one of tests will pass).
The minimal probability that you want is specified using minSuccessRate during the time window controlled by the timeLimit. Additionally if you specify too high minSuccessRate you’ll have too many retries, so the upper bound for this is controlled by the maxCount parameter so that this strategy will calculate the required number of retries according to the minSuccessRate but if it’s higher than the maxCount it will choose maxCount.
flakinessStrategy:
type: "probability"
minSuccessRate: 0.7
maxCount: 3
timeLimit: "2015-03-14T09:26:53.590Z"
marathon {
flakinessStrategy {
probabilityBased {
minSuccessRate = 0.7
maxCount = 3
timeLimit = Instant.now().minus(Duration.parse("PT1H"))
}
}
}
marathon {
flakinessStrategy {
probabilityBased {
minSuccessRate = 0.7
maxCount = 3
timeLimit = Instant.now().minus(Duration.parse("PT1H"))
}
}
}
This is the logic that kicks in if our preventive logic failed to anticipate such high number of retries. This works after the tests were actually executed.
As the name implies, no retries are done. This is the default mode.
retryStrategy:
type: "no-retry"
marathon {
retryStrategy {}
}
marathon {
retryStrategy {}
}
Parameter totalAllowedRetryQuota specifies how many retries at all (for all the tests is total) are allowed. retryPerTestQuota controls how many retries can be done for each test individually.
retryStrategy:
type: "fixed-quota"
totalAllowedRetryQuota: 100
retryPerTestQuota: 3
marathon {
retryStrategy {
fixedQuota {
retryPerTestQuota = 3
totalAllowedRetryQuota = 100
}
}
}
marathon {
retryStrategy {
fixedQuota {
retryPerTestQuota = 3
totalAllowedRetryQuota = 100
}
}
}
Filtering of tests is important since usually we as developers have the same codebase for all the different types of tests we want to execute. In order to indicate to marathon which tests you want to execute you can use the allowlist and blocklist parameters. First allowlist is applied, then the blocklist. Each accepts a TestFilter based on the class name, fully qualified class name, package, annotation or method. Each expects a regular expression as a value.
In order to filter using multiple filters at the same time a composition filter is also available which accepts a list of base filters and also an operation such as UNION, INTERSECTION or SUBTRACT. You can create complex filters such as get all the tests starting with E2E but get only methods from there ending with Test. Composition filter is not supported by groovy gradle scripts, but is supported if you use gradle kts.
An important thing to mention is that by default platform specific ignore options are not taken into account. This is because a cross-platform test runner cannot account for all the possible test frameworks out there. However, each framework’s ignore option can still be “explained” to marathon, e.g. JUnit’s org.junit.Ignore annotation can be specified in the filtering configuration.
filteringConfiguration:
allowlist:
- type: "simple-class-name"
regex: ".*"
- type: "fully-qualified-class-name"
regex: ".*"
- type: "method"
regex: ".*"
- type: "composition"
filters:
- type: "package"
regex: ".*"
- type: "method"
regex: ".*"
op: "UNION"
blocklist:
- type: "package"
regex: ".*"
- type: "annotation"
regex: ".*"
marathon {
filteringConfiguration {
allowlist {
simpleClassNameFilter = [".*"]
fullyQualifiedClassnameFilter = [".*"]
testMethodFilter = [".*"]
}
blocklist {
testPackageFilter = [".*"]
annotationFilter = [".*"]
}
}
}
marathon {
filteringConfiguration {
allowlist = listOf(
SimpleClassnameFilter(".*".toRegex()),
FullyQualifiedClassnameFilter(".*".toRegex()),
TestMethodFilter(".*".toRegex()),
CompositionFilter(
listOf(
TestPackageFilter(".*".toRegex()),
TestMethodFilter(".*".toRegex())
),
CompositionFilter.OPERATION.UNION
)
)
blocklist = listOf(
TestPackageFilter(".*".toRegex()),
AnnotationFilter(".*".toRegex())
)
}
}
By default, test classes are found using the "^((?!Abstract).)*Test[s]*$"
regex. You can override this if you need to.
testClassRegexes:
- "^((?!Abstract).)*Test[s]*$"
marathon {
testClassRegexes = [
"^((?!Abstract).)*Test[s]*\$"
]
}
marathon {
testClassRegexes = listOf(
"^((?!Abstract).)*Test[s]*$"
)
}
By default, the build fails if some tests failed. If you want to the build to succeed even if some tests failed use true.
ignoreFailures: true
marathon {
ignoreFailures = true
}
marathon {
ignoreFailures = true
}
This parameter specifies the behaviour for the underlying test executor to timeout if there is no output. By default, this is set to 60 seconds.
testOutputTimeoutMillis: 30000
marathon {
testOutputTimeoutMillis = 30000
}
marathon {
testOutputTimeoutMillis = 30000
}
This parameter specifies the behaviour for the underlying test executor to timeout if the batch execution exceeded some duration. By default, this is set to 15 minutes.
testBatchTimeoutMillis: 900000
marathon {
testBatchTimeoutMillis = 900000
}
marathon {
testBatchTimeoutMillis = 900000
}
When the test run starts device provider is expected to provide some devices. This should not take more than 3 minutes by default. If your setup requires this to be changed please override as following:
deviceInitializationTimeoutMillis: 300000
marathon {
deviceInitializationTimeoutMillis = 300000
}
marathon {
deviceInitializationTimeoutMillis = 300000
}
To better understand the use-cases that marathon is used for we’re asking you to provide us with anonymised information about your usage. By default, this is disabled. Use true to enable.
analyticsTracking: true
marathon {
analyticsTracking = true
}
marathon {
analyticsTracking = true
}
By default, tests that don’t have any status reported after execution (for example a device disconnected during the execution) retry indefinitely. You can limit the number of total execution for such cases using this option.
uncompletedTestRetryQuota: 100
marathon {
uncompletedTestRetryQuota = 100
}
marathon {
uncompletedTestRetryQuota = 100
}
By default, if one of the test retries succeeds then the test is considered successfully executed. If you require success status only when all retries were executed successfully you can enable the strict mode. This may be useful to verify that flakiness of tests was fixed for example.
strictMode: true
marathon {
strictMode = true
}
marathon {
strictMode = true
}
Enabled very verbose logging to stdout of all the marathon components. Very useful for debugging.
debug: true
marathon {
debug = true
}
marathon {
debug = true
}
By default, screen recording will only be pulled for tests that failed (ON_FAILURE option). This is to save space and also to reduce the test duration time since we’re not pulling additional files. If you need to save screen recording regardless of the test pass/failure please use the ON_ANY option:
screenRecordingPolicy: "ON_ANY"
marathon {
screenRecordingPolicy = com.malinskiy.marathon.execution.policy.ScreenRecordingPolicy.ON_ANY
}
marathon {
screenRecordingPolicy = com.malinskiy.marathon.execution.policy.ScreenRecordingPolicy.ON_ANY
}