download-asset

• We have users on macOS, Windows, and Linux.
• We have a blend of worker laptops using both Intel/AMD and Apple Silicon CPU architectures.
• We have cloud servers in AWS, GCP, Azure, and Oracle Cloud.
• We have a blend of user machines using both Intel/AMD and Apple Silicon CPU architectures.
• We rely heavily on Docker/Terraform/OpenTofu for consistency/repeatability, and to better scale the perpetually-limited resources of our DevOps/SRE/Cloud/Platform engineering teams.
• Docker runs natively on Linux.
• Docker runs virtualized in macOS and Windows.
• Software running inside the Linux-based Docker containers is most efficient when compiled for the current CPU architecture.
• Out on the internet, people build packages that can be installed. Many are not inside the Linux system’s package manager, and must be installed from the web. The people who publish these packages use a variety of identifiers for Intel-compatible vs ARM-compatible CPU architectures. There is no consistency.

When building tooling/solutions for a heterogenous set of machines across an enterprise, you need to solve for (at least) the following matrix.

• Current OS
• Current CPU architecture
• Package filenames on the internet

Deploying software as Docker containers (running Linux) helps normalize things like:

• Relying on GNU vs BSD-flavored CLI tools
• Download packages into the Docker container, worrying only about Linux
• Deploying software across worker laptops running different host operating systems
• Deploying software to Linux servers in the cloud

But these solutions don't solve the (relatively new) problem of an uptick of 64-bit ARM software/CPUs being added to the matrix — and the fact that these are not referred-to in a unified, consistent way.

Common values for uname -m

• ¹ Red Hat family includes Red Hat Enterprise Linux, CentOS, Fedora, Amazon Linux, and others.
• ² Debian family includes Debian, Ubuntu, Linux Mint, and others.
• ³ Busybox family includes Busybox, Alpine Linux, and others.
• ⁴ Windows WSL2 returns whatever the underlying Linux installation says.

Here's an (extremely) brief overview of modern CPU architectures that you most commonly find in cloud service providers and modern desktops/laptops.

This is meant to be illustrative, not comprehensive. As of today, these are the top 2 by a large margin.

download-asset get

This is the binary, and the subcommand get. Use the --help flag to get more information about additional options.

--owner-repo

Since we (at the moment) only support GitHub releases, this is the owner/repository pattern. In this example, we're going to download from aquasecurity/trivy.

--tag v0.49.1

Here, we've specified a tag in the repository. Since Assets can only be attached to Releases, this MUST be a Tag that has a Release attached to it. If you just want to grab the latest release (i.e., the release that is flagged as latest, not necessarily the highest version number), then you can either set --tag latest, or omit the flag all-together.

It will try the tag with a prepended v, then without a prepended v, and will respond if either of them match. If the tag doesn't exist, or follows a different format, download-asset will throw an error.

--linux Linux

This flag only applies when the current system is a Linux system. The same is true for the --darwin, --windows, --freebsd, and other OS-specific flags. If the current system is Linux (linux), then this is the string to use for the {{.OS}} value in the --pattern tag (more in a moment.)

You should set this for each OS you plan to download assets for, with the values matching the strings in the list of assets.

--darwin macOS \
--linux Linux \
--windows windows \
--freebsd FreeBSD \
--netbsd NetBSD \

--arm64 ARM64

This flag only applies when the current CPU architecture is 64-bit ARM. The same is true for the --arm32, --intel64, --intel32, and other CPU architecture-specific flags. If the current system is 64-bit ARM (arm64), then this is the string to use for the {{.Arch}} value in the --pattern tag (more in a moment.)

You should set this for each CPU architecture you plan to download assets for, with the values matching the strings in the list of assets.

--arm32 ARM \
--arm64 ARM64 \
--intel32 32bit \
--intel64 64bit \
--ppc32 PPC \
--ppc64 PPC64 \
--s390x s390x \

[!CAUTION] At the moment, there is not a good way to narrow focus in CPU architectures better than what is already implemented. For example, there is not a good way to discern between 32-bit ARMv6 and 32-bit ARMv7 — it's simply 32-bit ARM. We anticipate that this is good enough for most people. CPU architectures can be hard.

--pattern 'trivy_{{.Ver}}_{{.OS}}-{{.Arch}}.{{.Ext}}$'

This is the naming pattern to match when looking through the list of Assets attached to the Release. We already talked about the .OS and .Arch values, above.

The .Ver value is the tag (or the tag resolved when we selected latest) WITHOUT the prepended v. If the Asset name contains a v before the version, you should add the v directly in the --pattern value.

The .Ext value is a regular expression that matches most common archive file extensions (e.g., 7z, xz, tar.gz, tgz, tar.bz2, tbz2, zip) WITHOUT the preceding ..

Since this is a regular expression, the $ at the end means end of the string. This helps you avoid matches for Linux-ARM64.tar.gz.sig or windows-64bit.zip.pem since this tool will download the first match it finds. In order to ensure you get what you want, you are advised to make your pattern as specific as possible.

If your pattern (after resolving for .Ver, .OS, .Arch, and .Ext) is not a valid Go regular expression pattern, the app will panic and exit. The regular expression is passed through regexp.MustCompile.

--archive-path trivy

This is the path inside of the archive. In the case of Trivy, the trivy binary is in the root of the archive, and is named trivy.

In the case of golangci/golangci-lint v1.56.2 for linux/arm64, the path inside the archive is golangci-lint-1.56.2-linux-arm64/golangci-lint. You can use the same variables in --archive-path that you can in --pattern (.Ver, .OS, .Arch, and .Ext).

--archive-path 'golangci-lint-{{.Ver}}-{{.OS}}-{{.Arch}}/golangci-lint'

--write-to-bin trivy

This is the name to give to the binary when it's installed on your $PATH. Indownload-asset will attempt to install to /usr/local/bin by default. If it does not have permission, it will install to $HOME/bin.

As a result, in this example, download-asset will try to extract the trivy binary from the archive, and install it to /usr/local/bin/trivy. If that location is not writable, it will try to install to $HOME/bin/trivy. If it cannot write there, it will fail.

1. The GITHUB_TOKEN environment variable should be generated from your GitHub Enterprise Server instance, not public GitHub.com.

2. If your instance has Subdomain Isolation enabled, then your --endpoint flag is likely going to be api.github.company.com. Without subdomain isolation, it will likely be github.company.com. If you're not sure, ask your GitHub Enterprise Server administrators.

3. For --endpoint, pass the scheme+hostname (e.g., https://github.company.com or https://api.github.company.com). If your instance is running over insecure HTTP (port 80), specify http://. If you do not specify a scheme (e.g., api.github.company.com), then download-asset will assume HTTPS.

4. Keep in mind that the --owner-repo flag will refer to your organization's GitHub Enterprise Server environment, and NOT public GitHub.com.

For this project, we have to re-map most of the values to different spellings/formats.

[aquasecurity.trivy]
pattern      = "trivy_{{.Ver}}_{{.OS}}-{{.Arch}}.{{.Ext}}$"
archive-path = "trivy"
write-to-bin = "trivy"
darwin       = "macOS"
freebsd      = "FreeBSD"
linux        = "Linux"
windows      = "windows"
arm32        = "ARM"
arm64        = "ARM64"
intel32      = "32bit"
intel64      = "64bit"
ppc64le      = "PPC64LE"

For this project, they use all the standard naming (e.g., linux, arm64). The quirk here is that they don't archive their binaries in .zip or .tar.gz first. They just upload the binaries themselves directly to the release assets.

As a result, archive-path is an empty string.

[gruntwork-io.terragrunt]
pattern      = "terragrunt_{{.OS}}_{{.Arch}}$"
archive-path = ""
write-to-bin = "terragrunt"

For this project, they also upload raw binaries without archiving them. But they capitalize the first letter of the OS name, and chose to use the Red Hat version of the name for 64-bit Intel-compatible CPUs (x86_64 instead of amd64), and the Debian version of the name for 64-bit ARM CPUs (arm64 instead of aarch64).

[hadolint.hadolint]
pattern      = "hadolint-{{.OS}}-{{.Arch}}$"
archive-path = ""
write-to-bin = "hadolint"
darwin       = "Darwin"
linux        = "Linux"
windows      = "Windows"
intel64      = "x86_64"

For this project, they bundle their binary inside an archive, but it's in a subdirectory that has release information in the path name. For that reason, we use the .Ver variable to find the correct value inside the archive so that we can extract it.

[koalaman.shellcheck]
pattern      = "shellcheck-v{{.Ver}}.{{.OS}}.{{.Arch}}.{{.Ext}}$"
archive-path = "shellcheck-v{{.Ver}}/shellcheck"
write-to-bin = "shellcheck"
arm32        = "armv6hf"
arm64        = "aarch64"
intel64      = "x86_64"

download-asset’s .Ext variable can match assets with the following file extensions:

• exe
• tar.bz2
• tar.gz
• tar.xz
• tbz2
• tgz
• txz
• zip

And it can decode/read the following archive formats:

• tar + bzip2
• tar + gzip
• tar + xz
• zip

Others can be requested if we have a real-world repository to test against.