migrator

package
v0.0.0-...-e560ebb Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 13, 2021 License: BSD-3-Clause Imports: 13 Imported by: 0

Documentation

Overview

Package migrator provides interfaces and tooling for migrating LUCI configuration files across all known LUCI projects.

The tool revolves around the user providing a plugin to perform bulk analysis and optional fixes across all LUCI projects' configuration folders, from the point of view of the 'luci-config' service.

The plugin currently has two main points of extension:

  • FindProblems - Analyze the LUCI project and config file contents to see if the project has migrated. This can inspect the project name, as well as the presence and contents of any config files.
  • ApplyFix - If FindProblems revealed issues, ApplyFix will be run in the context of a local sparse+shallow checkout containing the configuration files. This has the ability to run programs in the checkout, as well as stat/read/modify files.

This package contains the interface definitions for the migrator plugin.

Index

Constants

View Source
const TieStderr = "2>&1"

TieStderr is a special token for Shell.{Run,Retval,Stdout} to indicate that Stderr content should be redirected to Stdout.

Variables

This section is empty.

Functions

func NonActionable

func NonActionable(r *Report)

NonActionable is-a ReportOption which indicates that this Report cannot be fixed by ApplyFix. If there are no Actionable Reports for a given project in FindProblems, the checkout and ApplyFix phase will be skipped.

Types

type API

type API interface {
	// FindProblems allows you to report problems about a Project, or about
	// certain configuration files within the project.
	//
	// If the method finds issues which warrant followup, it should use
	// proj.Report and/or proj.ConfigFiles()["filename"].Report. Reporting one or
	// more problems will cause the migrator tool to set up a checkout for this
	// project.
	//
	// Logging is set up for this context, and will be diverted to a per-project
	// logfile.
	//
	// `proj` is implemented with LUCI Config API calls; it reflects the state of
	// the project as currently known by the luci-config service.
	//
	// This function should panic on error.
	FindProblems(ctx context.Context, proj Project)

	// ApplyFix allows you to attempt to automatically fix problems within a repo.
	//
	// Note that for real implementations you may want to keep details on the
	// `impl` struct; this will let you carry over information from
	// FindProblems.
	//
	// Logging is set up for this context, and will be diverted to a per-project
	// logfile.
	//
	// `repo.Project()` is implemented with local filesystem interactions on the
	// checked-out repo. This may differ from the current state of the luci-config
	// service.
	//
	// This function should panic on error.
	ApplyFix(ctx context.Context, repo Repo)
}

API is the implementation of a plugin, as returned by the plugin's InstantiateAPI function.

One API instance will be created per LUCI project during 'scan', and `FindProblems` will be invoked before `ApplyFix`.

type ConfigFile

type ConfigFile interface {
	Reportable

	// Path returns the path relative to the repo's generated configuration
	// directory.
	Path() string

	// RawData returns the data of this configuration file as a string.
	//
	// May do a (cached) network operation to retrieve the raw data.
	RawData() string

	// TextPb parses the raw data as a text proto with "heredoc" processing into
	// `out`.
	//
	// May do a (cached) network operation to retrieve the raw data.
	//
	// If there are errors during parsing, this panics.
	TextPb(out proto.Message)
}

ConfigFile encapsulates as single configuration file from a Project.

type InstantiateAPI

type InstantiateAPI func() API

InstantiateAPI is the symbol that plugins must export.

It should return a new instance of API.

If this returns nil, it has the effect of a plugin which:

FindProblems reports a generic problem "FindProblems not defined".
ApplyFix does nothing.

type Project

type Project interface {
	Reportable

	ID() string

	// ConfigFiles returns a mapping of path to config file.
	//
	// May do a (cached) network operation to retrieve the data.
	ConfigFiles() map[string]ConfigFile
}

Project encapsulates the pertinent details of a single LUCI Project's configuration files.

type Repo

type Repo interface {
	// ConfigRoot returns the path to the config file 'root'.
	//
	// This would be the directory containing `main.star`, if the repo has one,
	// otherwise is identical to GeneratedConfigRoot.
	//
	// TODO: Have a less-heuristic way to find this path in the repo.
	//
	// This an 'absolute-style' path (see Shell).
	ConfigRoot() string

	// GeneratedConfigRoot returns the path to the generated config files (i.e.
	// the ones seen by the luci-config service).
	//
	// This an 'absolute-style' path (see Shell).
	GeneratedConfigRoot() string

	// Project returns the LUCI Project associated with this repo.
	//
	// Files retrieved with ConfigFiles() will be based on the checked-out data.
	Project() Project

	// Shell returns a new shell object for this repo with its current working
	// directory set to ConfigRoot().
	Shell() Shell
}

Repo represents a checked-out git repo on disk.

Use Shell to manipulate the repo in your plugin's ApplyFix method.

type Report

type Report struct {
	ReportID

	Tag     string
	Problem string

	// If true, indicates that this report can be fixed by ApplyFix.
	Actionable bool

	Metadata map[string]stringset.Set
}

Report stores a single tagged problem (and metadata).

func NewReportFromCSVRow

func NewReportFromCSVRow(row []string) (ret *Report, err error)

NewReportFromCSVRow creates a new Report from a CSVRow written with ToCSVRow.

func (*Report) Clone

func (r *Report) Clone() *Report

Clone returns a deep copy of this Report.

func (*Report) ToCSVRow

func (r *Report) ToCSVRow() []string

ToCSVRow returns a CSV row:

Project, ConfigFile, Tag, Problem, Actionable, Metadata*

Where Metadata* is one key:value entry per value in Metadata.

type ReportDump

type ReportDump struct {
	// contains filtered or unexported fields
}

ReportDump is a mapping of all reports, generated via DumpReports(ctx).

It maps the ReportID to a list of all Reports found for that ReportID.

func NewReportDumpFromCSV

func NewReportDumpFromCSV(raw io.Reader) (*ReportDump, error)

NewReportDumpFromCSV reads a raw CSV and returns the *ReportDump.

CSV must be a compatible schema version (i.e. produced with a similar version of ReportDump.WriteToCSV.

func (*ReportDump) Add

func (r *ReportDump) Add(reports ...*Report)

Add appends one or more reports to this ReportDump.

func (*ReportDump) Clone

func (r *ReportDump) Clone() *ReportDump

Clone makes a deep copy of this ReportDump.

func (*ReportDump) Empty

func (r *ReportDump) Empty() bool

Empty returns true iff this ReportDump has no entries.

func (*ReportDump) Iterate

func (r *ReportDump) Iterate(cb func(ReportID, []*Report) bool)

Iterate invokes `cb` for each ReportID with all Reports from that ReportID.

`cb` will be called in sorted order on ReportID. If it returns `false`, iteration will stop.

func (*ReportDump) UpdateFrom

func (r *ReportDump) UpdateFrom(other *ReportDump) int

UpdateFrom appends `other` to this ReportDump.

Returns the number of Report records in `other`.

func (*ReportDump) WriteToCSV

func (r *ReportDump) WriteToCSV(out io.Writer) error

WriteToCSV writes this ReportDump out as CSV.

type ReportID

type ReportID struct {
	Project    string
	ConfigFile string
}

ReportID is a simple Project/ConfigFile tuple and identifies the object which generated the report.

func (ReportID) ConfigSet

func (r ReportID) ConfigSet() config.Set

ConfigSet returns the luci-config "config.Set" for this report.

e.g. "projects/${Project}"

func (ReportID) String

func (r ReportID) String() string

type ReportOption

type ReportOption func(*Report)

ReportOption allows attaching additional optional data to reports.

func MetadataOption

func MetadataOption(key string, values ...string) ReportOption

MetadataOption returns a ReportOption which allows attaching a string-string multimap of metadatadata to a Report.

type Reportable

type Reportable interface {
	// Report logs a problem about this object.
	//
	// `tag` should be a CAPS_STRING which makes sense to your particular
	// application.
	//
	// `description` should be a human readable explanation of the problem.
	//
	// Example:
	//    tag: MISSING_BLARF
	//    description: "blarf.cfg file is missing"
	Report(tag, description string, opts ...ReportOption)
}

Reportable is an interface for reporting problems about an object.

Implemented by Project and ConfigFile.

type Shell

type Shell interface {
	// Cd changes the current directory.
	//
	// Absolute paths (i.e. beginning with '/') are interpreted as relative to the
	// repo root.
	//
	// Attempting to Cd out of the top of the repo is an error.
	Cd(path string)

	// ModifyFile allows trivial modification of a file.
	//
	// This will call `modify` with the contents of the old file ("" if the file
	// didn't exist), and will write the returned string back to `path`, if the
	// returned string is different. This will create missing intermediate
	// directories.
	//
	// Allows supplying FileMode. If omitted, defaults to 0666 (i.e. a+rw) for new
	// files, and otherwise will keep the mode of the existing file.
	ModifyFile(path string, modify func(oldContents string) string, mode ...os.FileMode)

	// Stat returns the FileInfo for the entity at `path`, or nil if no such
	// entity exists.
	Stat(path string) os.FileInfo

	// Run executes a command `name` with arguments `args`.
	//
	// `name` may be relative to the Shell's cwd (e.g. `./main.star`), or to $PATH
	// (e.g. `git`).
	//
	// The command is expected to return exitcode 0.
	//
	// The command runs with the cwd of the Shell.
	//
	// Stdout+Stderr are redirected to logging (with Stdout logged at 'Info' level
	// and Stderr logged at 'Error' level).
	//
	// If the very last item in `args` is the TieStderr constant, both will be
	// tied together and all output will be logged at 'Info' level.
	Run(name string, args ...string)

	// Same as `Run`, but returns the exitcode of the process (instead of
	// asserting an exit code of 0).
	Retval(name string, args ...string) int

	// Same as `Run`, but returns the stdout of the process. If TieStderr is the
	// last argument, then this captures Stderr, too.
	Stdout(name string, args ...string) string
}

Shell is a basic interface for interacting with the Repo.

Paths

All `path` arguments to the Shell are either:

  • start with '/' and are relative to the corresponding Repo's root.
  • OR; are relative to the Shell's current working directory.

It is not permitted to access a path outside the Repo's root.

A just-created Shell starts with the 'cwd' at the ConfigDir.

Errors

All functions of Shell will panic under error conditions. This is consistent with the API of the ApplyFix plugin method.

Directories

Path Synopsis
cmd
internal
plugsupport
Package plugsupport provides implementations for loading migrator plugins.
Package plugsupport provides implementations for loading migrator plugins.
plugsupport/templates
Package templates is generated by go.chromium.org/luci/tools/cmd/assets.
Package templates is generated by go.chromium.org/luci/tools/cmd/assets.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL