[Flutter] Gitlab CI - flutter test running very slow

I've been managing a project for the last couple of months using Flutter and TDD on Gitlab and using the Gitlab CI for monitoring code quality and tests. With the end of the project tin sight we now have over 700 tests and our CI has reached glacial speeds. At first we would take 5-10min to run the whole pipe, now it can take as long as 58min, with the only noticeable difference being the number of tests.

After removing the --machine from our scripts I noticed that the flutter test --coverage is taking 11 times longer than when run locally.

LOCAL

PS C:\Users\lr\Documents\GitHub\hive-manager> flutter test --coverage
04:04 +707: All tests passed!


GITLAB CI

$ flutter test --coverage
45:30 +707: All tests passed!


It is definitely the test phase of the pipeline that is causing issue as when I look at each jobs time there is a noticeable difference:

code_quality - 00:05:24
test - 00:48:47
coverage - 00:02:06 (82.2%)
semantic-version - 00:01:20


I'm a little lost on what to do now as I still have more tests but the CI is starting to cost quite a bit in minutes while at the same time I don't want to have to rebuild the CI somewhere else. Is this just a limit of Gitlab CI or is something going wrong here. I've attached the .yaml below

stages: # List of stages for jobs, and their order of execution
- analyze
- test
- coverage
- semantic-version

default:
image: cirrusci/flutter:latest

cache:
paths:
- /flutter/bin/cache/dart-sdk

code_quality:
stage: analyze
before_script:
- pub global activate dart_code_metrics
- export PATH="$PATH":"$HOME/.pub-cache/bin"
script:
- flutter --version
- flutter analyze
- metrics lib -r codeclimate > gl-code-quality-report.json
rules:
- if: $CI_COMMIT_BRANCH == "main"
- if: $CI_COMMIT_BRANCH == "develop" && $CI_PIPELINE_SOURCE == "push"
- if: $CI_COMMIT_BRANCH == "develop" && $CI_PIPELINE_SOURCE == "merge_request_event"
allow_failure: true

artifacts:
reports:
codequality: gl-code-quality-report.json

test: # This job runs in the test stage.
stage: test # It only starts when the job in the build stage completes successfully.
script:
- flutter test --coverage
rules:
- if: $CI_COMMIT_BRANCH == "main"
- if: $CI_COMMIT_BRANCH == "develop" && $CI_PIPELINE_SOURCE == "push"
- if: $CI_COMMIT_BRANCH == "develop" && $CI_PIPELINE_SOURCE == "merge_request_event"

coverage: # This job runs in the test stage.
stage: coverage # It only starts when the job in the build stage completes successfully.
script:
- lcov --summary coverage/lcov.info
- lcov --remove coverage/lcov.info
"lib/config/*"
"lib/application/l10n/l10n.dart"
"lib/application/l10n/**/*"
"lib/domain/repositories/*"
"lib/injection.config.dart"
"lib/presentation/routes/*"
"lib/infrastructure/repositories/firebase_injectable_module.dart"
"**/mock_*.dart"
"**/*.g.dart"
"**/*.gr.dart"
"**/*.freezed.dart"
"**/*.mocks.dart"
"**/*.config.dart"
-o coverage/clean_lcov.info
- genhtml coverage/clean_lcov.info --output=coverage
- curl -Os https://uploader.codecov.io/latest/linux/codecov
- chmod +x codecov
- ./codecov -t $CODECOV_TOKEN
- mv coverage/ public/
rules:
- if: $CI_COMMIT_BRANCH == "main"
- if: $CI_COMMIT_BRANCH == "develop" && $CI_PIPELINE_SOURCE == "push"
- if: $CI_COMMIT_BRANCH == "develop" && $CI_PIPELINE_SOURCE == "merge_request_event"

coverage: '/lines\.*: \d+\.\d+\%/'
artifacts:
paths:
- public

semantic-version:
image: node:16
stage: semantic-version
only:
refs:
- main
- develop
script:
- touch CHANGELOG.md
- npm install @semantic-release/gitlab @semantic-release/changelog
- npx semantic-release
artifacts:
paths:
- CHANGELOG.md


ADDITIONAL INFO Looking into the report.xml from GitLab I can also see that most tests take ~2sec to complete, but the total tests only take 135sec.

While the test job takes ~44min to complete.

I've also tried removing the --coverage to reduce the time which resulted in 8min30sec, but which is still a lot more than the 1min30sec that it takes locally.

Continue reading...