run-terraform-schema

What is this?

run-terraform-schema downloads, describes, and manages tasks related to terraform provider schemas.

When should it be used?

Whenever the terraform provider schema changes, specifically for Dataflow related resources. Therefore, usage should be infrequent.

Why was it made?

The purpose of run-terraform-schema is to generate artifacts that are dependencies to autogenerate terraform modules. While run-terraform-schema is intended for infrequent use, the Dataflow related resources have numerous arguments making the generation of said artifact dependencies error-prone when done manually.

How does it work?

All of its contained subcommands are intended for use as a pipeline, summarized in the diagram below.

First a user runs the download subcommand to pull the latest registry schema from the Hashicorp provider registry, saving the JSON output to a temporary location. Next a user runs the describe subcommand twice for classic and flex templates to filter the raw output from the previous step. The output of running describe is version controlled at cicd/internal/terraform/schemas. Finally, a user runs the generate freemarker subcommand, which defaults to using cicd/internal/terraform/schemas as input, to generate the freemarker used to autogenerate terraform modules.

---
title: go run ./cmd/run-terraform-schema
---
flowchart TD
%% Nodes
download["`download`"]
provider["provider schema output:\n/path/to/temporary.json"]
generate_google_dataflow_job["`generate freemarker google google_dataflow_job`"]
generate_google_dataflow_flex_template_job["`generate freemarker google-beta google_dataflow_flex_template_job`"]
google_dataflow_job["`describe google google_dataflow_job`"]
google_dataflow_flex_template_job["`describe google-beta google_dataflow_flex_template_job`"]
google_dataflow_job_resource["cicd/internal/terraform/schemas/google.json:\n\n #123; #quot;google_dataflow_job#quot;: ...#125;"]
google_dataflow_flex_template_job_resource["cicd/internal/terraform/schemas/google-beta.json:\n\n #123; #quot;google_dataflow_flex_template_job#quot;: ...#125;"]
freemarker_classic_template["plugins/core-plugin/src/main/resources/terraform-classic-template.tf"]
freemarker_flex_template["plugins/core-plugin/src/main/resources/terraform-flex-template.tf"]

%% Interactions
click freemarker_classic_template href "https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/main/plugins/core-plugin/src/main/resources/terraform-classic-template.tf" "plugins/core-plugin/src/main/resources/terraform-classic-template.tf" _blank
click freemarker_flex_template href "https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/main/plugins/core-plugin/src/main/resources/terraform-flex-template.tf" "plugins/core-plugin/src/main/resources/terraform-flex-template.tf" _blank
click google_dataflow_job_resource href "https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/main/cicd/internal/terraform/schemas/google.json" "cicd/internal/terraform/schemas/google.json" _blank
click google_dataflow_flex_template_job_resource href "https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/main/cicd/internal/terraform/schemas/google-beta.json" "cicd/internal/terraform/schemas/google-beta.json" _blank

%% Paths
download --> provider
provider --> google_dataflow_job
provider --> google_dataflow_flex_template_job

subgraph classic ["Generate Classic Template Artifacts"]
    google_dataflow_job --> google_dataflow_job_resource
    google_dataflow_job_resource --> generate_google_dataflow_job
    generate_google_dataflow_job --> freemarker_classic_template
end

subgraph flex ["Generate Flex Template Artifacts"]
    google_dataflow_flex_template_job --> google_dataflow_flex_template_job_resource
    google_dataflow_flex_template_job_resource --> generate_google_dataflow_flex_template_job
    generate_google_dataflow_flex_template_job --> freemarker_flex_template
end

How to use it?

During the infrequent occurrence of Google or Google-Beta provider Dataflow related resource changes, the following steps illustrate how to generate important artifacts to autogenerate terraform modules.

Step 1. Download the provider schemas

Run the following to download the provider schemas to a temporary location:

go run ./cmd/run-terraform-schema download /path/to/temporary.json

Step 2a. Generate Classic Template Resource Schema and Freemarker Resource

Run the describe subcommand to filter the temporary output and store in cicd/internal/terraform/schemas/google.json

go run ./cmd/run-terraform-schema describe google google_dataflow_job /path/to/temporary.json > internal/terraform/schemas/google.json

Finally, run the generate freemarker subcommand and store in plugins/core-plugin/src/main/resources/terraform-classic-template.tf

go run ./cmd/run-terraform-schema generate freemarker google google_dataflow_job ../plugins/core-plugin/src/main/resources/terraform-classic-template.tf

Step 2b. Generate Flex Template Resource Schema and Freemarker Resource

Run the describe subcommand to filter the temporary output and store in cicd/internal/terraform/schemas/google-beta.json

go run ./cmd/run-terraform-schema describe google-beta google_dataflow_flex_template_job /path/to/temporary.json > internal/terraform/schemas/google-beta.json

Finally, run the generate freemarker subcommand and store in plugins/core-plugin/src/main/resources/terraform-flex-template.tf

go run ./cmd/run-terraform-schema generate freemarker google-beta google_dataflow_flex_template_job ../plugins/core-plugin/src/main/resources/terraform-flex-template.tf

Getting Help

To show help, run:

go run ./cmd/run-terraform-schema

You should see the following output:

Downloads, describes, and manages tasks related to terraform provider schemas

Usage:
run-terraform-schema [command]

Available Commands:
completion  Generate the autocompletion script for the specified shell
describe    Describe terraform provider schemas.
download    Acquires the schema of terraform providers.
Outputs to FILE, created if needed; outputs to STDOUT if no FILE provided.
generate    Generates content based on terraform provider schemas
help        Help about any command

Flags:
-h, --help   help for run-terraform-schema

Use "run-terraform-schema [command] --help" for more information about a command.