How to perftest a model
=======================
For each model running inside Firefox, we want to determine its performance
in terms of speed and memory usage and track it over time.
To do so, we use the `Perfherder <
https://wiki.mozilla.org/Perfherder>`_ infrastructure
to gather the performance metrics.
Adding a new performance test is done in two steps:
1. make it work locally
2. add it in perfherder
Run locally
-----------
To test the performance of a model, you can add in the `tests/browser` a new file
with the following structure and adapt it to your needs:
.. code-block:: javascript
"use strict";
// unfortunately we have to write a full static structure here
// see
https://bugzilla.mozilla.org/show_bug.cgi?id=1930955
const perfMetadata = {
owner: "GenAI Team",
name: "ML Test Model",
description: "Template test for latency for ml models",
options: {
default: {
perfherder: true,
perfherder_metrics: [
{ name: "pipeline-ready-latency", unit: "ms", shouldAlert: true },
{ name: "initialization-latency", unit: "ms", shouldAlert: true },
{ name: "model-run-latency", unit: "ms", shouldAlert: true },
{ name: "pipeline-ready-memory", unit: "MB", shouldAlert: true },
{ name: "initialization-memory", unit: "MB", shouldAlert: true },
{ name: "model-run-memory", unit: "MB", shouldAlert: true },
{ name: "total-memory-usage", unit: "MB", shouldAlert: true },
],
verbose: true,
manifest: "perftest.toml",
manifest_flavor: "browser-chrome",
try_platform: ["linux", "mac", "win"],
},
},
};
requestLongerTimeout(120);
add_task(async function test_ml_generic_pipeline() {
const options = {
taskName: "feature-extraction",
modelId: "Xenova/all-MiniLM-L6-v2",
modelHubUrlTemplate: "{model}/{revision}",
modelRevision: "main",
};
const args = ["The quick brown fox jumps over the lazy dog."];
await perfTest("example", options, args);
});
Then add the file in `perftest.toml` and rebuild with `./mach build`.
The test downloads models it uses from the local disk, so you need to prepare them.
- Create a directory with a subdirectory called `onnx-models`.
- Download all the models in the subdirectory
The directory follows a `organization/name/revision` structure.
To make the previous example work, it would require you to download
the model files locally under `<ROOT>/onnx-models/Xenova/all-MiniLM-L6-v2/main`
Example:
.. code-block:: bash
cd ROOT/onnx-models
git lfs install
git clone -b main
https://huggingface.co/Xenova/all-MiniLM-L6-v2 onnx-models/Xenova/a
ll-MiniLM-L6-v2/main/
Once done, you should then be able to run it locally with :
.. code-block:: bash
MOZ_FETCHES_DIR=/Users/tarekziade/Dev/fetches ./mach perftest toolkit/components/ml/tests/browser/browser_ml_engine_perf.js --mochitest-extra-args=headless
Notice that `MOZ_FETCHES_DIR` is an absolute path to the `root` directory.
Add in the CI
-------------
To add the test in the CI you need to add an entry in
- taskcluster/kinds/perftest/linux.yml
- taskcluster/kinds/perftest/windows11.yml
- taskcluster/kinds/perftest/macos.yml
With a unique name that starts with `ml-perf`
Example for Linux:
.. code-block:: yaml
ml-perf:
fetches:
fetch:
- ort.wasm
- ort.jsep.wasm
- ort-training.wasm
- xenova-all-minilm-l6-v2
description: Run ML Models Perf Tests
treeherder:
symbol: perftest(linux-ml-perf)
tier: 2
attributes:
batch: false
cron: false
run-on-projects: [autoland, mozilla-central]
run:
command: >-
mkdir -p $MOZ_FETCHES_DIR/../artifacts &&
cd $MOZ_FETCHES_DIR &&
python3 python/mozperftest/mozperftest/runner.py
--mochitest-binary ${MOZ_FETCHES_DIR}/firefox/firefox-bin
--flavor mochitest
--output $MOZ_FETCHES_DIR/../artifacts
toolkit/components/ml/tests/browser/browser_ml_engine_perf.js
You also need to add the models your test uses (like the ones you've downloaded locally) by adding entries in
`taskcluster/kinds/fetch/onnxruntime-web-fetch.yaml` and adapting the `fetches` section.
Once this is done, try it out with:
.. code-block:: bash
./mach try perf --single-run --full --artifact
You should then see the results in treeherder.
