Title here
Summary here
Autodiscovery
Description
In the lifecycle of a software project, dependencies are everywhere. From that third application used to lint our code, the tool used to build our documentation website, or the container image used to distribute our application. As the project grows, the number of dependencies grows as long. Some dependencies are context-dependent, and others are expressed using a standard data structure.
Updatecli is a great tool to handle both situations. It can either manage tailored update scenarios by using a manifest provided by a maintainer or automatically identify available update scenarios.
The latter approach removes the need to write a manifest. This behavior is referred to as "autodiscovery". With autodiscovery enabled, updatecli generates manifests in-memory before applying them (instead of reading manifests and applying).
A use case: a repository with a collection of Dockerfile`s.
To keep the Dockerfiles' `FROM instructions up to date, writing one manifest file per Dockerfile isn’t practical. With autodiscovery enabled, Updatecli scans the repository for all Dockerfile and automatically tracks the FROM instruction to propose updates if needed.
This helps us maintain our Dockerfile images without having to write manifests.
Each autodiscovery scenario is handled by a crawler.
The goal of the crawler is to recursively parse all files in a directory, looking for the pattern and then try to generate as many updatecli manifests as possible.
Once all manifests have been generated, we run them as we would do with custom manifests, by running updatecli diff or updatecli apply.
As with any opinionated way of working, a bit of adaptability is required. The next part of this document covers the different kinds of customization that can be used with autodiscovery.
Default crawlers
Autodiscovery works out of the box, with no configuration and no --experimental flag, by running
one of the following commands from the directory to scan:
updatecli pipeline diffupdatecli manifest showupdatecli pipeline apply
Important | The default crawlers only run when Updatecli finds no manifest at all, that is, no As soon as one manifest is loaded, Updatecli runs that manifest and nothing else.
To combine your own manifests with autodiscovery, declare the |
For example, in a directory holding a single Dockerfile with FROM golang:1.21,
updatecli manifest show generates:
name: 'deps(dockerfile): bump "golang" digest'
sources:
golang:
name: get latest image tag for "golang"
kind: dockerimage
spec:
image: golang
tagfilter: ^\d*(\.\d*){1}$
versionfilter:
kind: semver
pattern: '>=1.21'
golang-digest:
dependson:
- golang
name: get latest image "golang" digest
kind: dockerdigest
spec:
image: golang
tag: '{{ source "golang" }}'
targets:
golang:
name: 'deps: update Docker image "golang" to "{{ source "golang" }}"'
kind: dockerfile
spec:
file: Dockerfile
instruction:
keyword: FROM
matcher: golang
sourceid: golang-digestThe crawlers enabled by default are:
argocd, bazel, cargo, dockercompose, dockerfile, flux, github/action, golang,
helm, helmfile, ko, kubernetes, maven, nomad, npm, precommit, prow, pyproject,
rancher/fleet, terraform, terragrunt, updatecli, woodpecker.
Some crawlers accept aliases, which behave exactly like the canonical key: go and golang/gomod
for golang, python/uv for pyproject.
Two crawlers are never enabled by default and must be requested explicitly:
gitea/action, which currently behaves like github/action, and plugin, used for
.wasm crawlers, which requires the --experimental flag.
Parameters
| Name | Type | Description | Required |
|---|---|---|---|
| actionid | string | actionid is a unique identifier used to retrieve the action configuration from the configuration file. | |
| crawlers | object | Crawlers defines a map of crawler configuration where the key represent the crawler type | |
| argocd | object | Spec defines the parameters which can be provided to the argocd builder. More information on the argocd documentation page | |
| bazel | object | Spec defines the Bazel autodiscovery parameters. More information on the bazel documentation page | |
| cargo | object | Spec defines the Cargo parameters. More information on the cargo documentation page | |
| dockercompose | object | Spec is a struct fill from Updatecli manifest data and shouldn’t be modified at runtime unless For Fields that requires it, we can use the struct DockerCompose Spec defines the parameters which can be provided to the Helm builder. More information on the dockercompose documentation page | |
| dockerfile | object | Spec is a struct fill from Updatecli manifest data and shouldn’t be modified at runtime unless For Fields that requires it, we can use the struct Dockerfile Spec defines the parameters which can be provided to the Dockerfile crawler. More information on the dockerfile documentation page | |
| flux | object | Spec defines the parameters which can be provided to the Flux crawler. More information on the flux documentation page | |
| gitea/action | object | Spec defines the parameters which can be provided to the Action crawler. More information on the gitea/action documentation page | |
| github/action | object | Spec defines the parameters which can be provided to the Action crawler. More information on the github/action documentation page | |
| golang | object | Spec defines the parameters which can be provided to the Golang autodiscovery builder. More information on the golang documentation page | |
| helm | object | Spec defines the Helm parameters. More information on the helm documentation page | |
| helmfile | object | Spec defines the Helmfile parameters. More information on the helmfile documentation page | |
| ko | object | Spec defines the parameters which can be provided to the Kubernetes builder. More information on the ko documentation page | |
| kubernetes | object | Spec defines the parameters which can be provided to the Kubernetes builder. More information on the kubernetes documentation page | |
| maven | object | Spec defines the parameters which can be provided to the Helm builder. More information on the maven documentation page | |
| nomad | object | Spec is a struct fill from Updatecli manifest data and shouldn’t be modified at runtime unless For Fields that requires it, we can use the struct DockerCompose Spec defines the parameters which can be provided to the Helm builder. More information on the nomad documentation page | |
| npm | object | Spec defines the parameters which can be provided to the NPM builder. More information on the npm documentation page | |
| plugin | object | More information on the plugin documentation page | |
| precommit | object | Spec defines the parameters uses to generate the precomit manifests More information on the precommit documentation page | |
| prow | object | Spec defines the parameters which can be provided to the Kubernetes builder. More information on the prow documentation page | |
| pyproject | object | Spec defines the pyproject autodiscovery parameters. More information on the pyproject documentation page | |
| rancher/fleet | object | Spec defines the parameters which can be provided to the fleet builder. More information on the rancher/fleet documentation page | |
| terraform | object | Spec defines the Terraform parameters. More information on the terraform documentation page | |
| terragrunt | object | Spec defines the Terraform parameters. More information on the terragrunt documentation page | |
| updatecli | object | Spec defines the Updatecli parameters. More information on the updatecli documentation page | |
| woodpecker | object | Spec is a struct filled from Updatecli manifest data and shouldn’t be modified at runtime unless For Fields that requires it, we can use the struct Woodpecker Spec defines the parameters which can be provided to the Woodpecker autodiscovery plugin. More information on the woodpecker documentation page | |
| groupby | string | groupby specifies how to group pipeline. The Accepted is one of “all”, “individual”. Default is “all” default: all | |
| pullrequestid | string | !Deprecated in favor of actionid | |
| scmid | string | scmid is a unique identifier used to retrieve the scm configuration from the configuration file. |
Autodiscovery could benefit from some customization, such as providing SCM configuration or defining pull request information. Some parameters are crawler specific and others apply to all crawlers.
A manifest can drive autodiscovery itself, through the root key "autodiscovery", which is what makes it possible to point crawlers at a remote repository, to attach a pull request action, or to run only a subset of the crawlers. In the following example, we clone the epinio/helm-chart repository in a temporary location and then look in that location for all Helm charts that specify dependencies and we try to update them.
Example
scms:
default:
kind: git
spec:
url: https://github.com/epinio/helm-charts.git
autodiscovery:
scmid: default
crawlers:
helm:
By default, autodiscovery looks for patterns from the local directory. We can also specify manifests from remote git repositories.
Tip | Declaring an autodiscovery key selects exactly the crawlers listed under crawlers. The default list is not merged in, so a manifest with crawlers: {golang:} only runs the Golang crawler. Conversely, listing a crawler with an empty value, such as helm: above, runs it with its default settings. |
SCM
We assume that each crawler relies on the same SCM configuration and we can add more autodiscovery manifests to handle more repositories.
Inherited manifest settings
Pipelines generated by a crawler are not fully independent: they inherit part of the parent manifest.
| Key | Inherited behavior |
|---|---|
| Every label of the autodiscovery manifest is copied onto each generated pipeline, so |
| Generated pipelines reuse the parent |
| All the scms of the parent manifest are made available to the generated pipelines. |
| The action referenced by |
Note | When groupby is all and the action has no title, Updatecli reuses the manifest name as the pull request title. This matters, because leaving it empty would give each discovered pipeline a different title and rewrite the pull request title on every run. When the manifest has no name either, Updatecli falls back to the generic deps: bumping various version. |
Pull Request
You can configure Updatecli to create a single pull request that includes all
autodiscovered changes by adding the actionid key to the autodiscovery
section of your manifest. This key should reference a pull request action
defined in your manifest’s actions section.
Here’s an example configuration for a GitHub pull request action:
Example
autodiscovery:
groupby: all
scmid: github
actionid: pr
actions:
pr:
kind: github/pullrequest
scmid: github
spec:
title: Pull request title
scms:
github:
kind: "github"
spec:
user: "user"
email: "updatecli@example.org"
owner: "my-org"
repository: "my-repo"
token: '{{ requiredEnv "GITHUB_TOKEN" }}'
username: '{{ requiredEnv "GITHUB_AUTHOR" }}'
One Pull request or many
The attribute autodiscovery.groupby can have the values:
all(default) to create one pull request with all changes. All the generated pipelines reuse thepipelineidof the autodiscovery manifest, so they share one Git branch and one pull request.individualto create one pull request per change. Each generated pipeline gets its ownpipelineid, derived from the autodiscoverypipelineidand the pipeline name, hence its own branch and pull request.
Important | all is the deliberate default: it avoids accidentally opening dozens of pull requests in a single Updatecli run. Switch to individual only when you are ready for that volume. |
Deprecated keys
autodiscovery.pullrequestid is deprecated in favor of autodiscovery.actionid.
Updatecli still translates it, with a warning, but fails when both are set.
Crawlers
Crawlers implement common update scenarios.
You can read more about the different crawlers by browsing the Plugins → Autodiscovery section of the documentation.
Type
The purpose of a crawler is to generate a manifest that updatecli can use. We identify two kinds of crawler, "standard", and "custom".
standard
A "standard" crawler is enabled by default but can be disabled on a case by case basis.
custom
A "custom" crawler is context specific, such as tailored to a company or a OSS project. By default it isn’t enabled.
Prev
Git Commit