checker

package
v1.105.4 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: May 29, 2024 License: AGPL-3.0 Imports: 22 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source
var (
	// Error is a standard error class for this package.
	Error = errs.Class("repair checker")
)

Functions

This section is empty.

Types

type Config

type Config struct {
	Interval time.Duration `` /* 129-byte string literal not displayed */

	ReliabilityCacheStaleness time.Duration            `help:"how stale reliable node cache can be" releaseDefault:"5m" devDefault:"5m" testDefault:"1m"`
	RepairOverrides           RepairOverrides          `` /* 148-byte string literal not displayed */
	RepairThresholdOverrides  RepairThresholdOverrides `help:"comma-separated override values for repair threshold in the format k-threshold" releaseDefault:"29-52" devDefault:""`
	RepairTargetOverrides     RepairTargetOverrides    `help:"comma-separated override values for repair success target in the format k-target" releaseDefault:"29-65" devDefault:""`
	// Node failure rate is an estimation based on a 6 hour checker run interval (4 checker iterations per day), a network of about 9200 nodes, and about 2 nodes churning per day.
	// This results in `2/9200/4 = 0.00005435` being the probability of any single node going down in the interval of one checker iteration.
	NodeFailureRate            float64  `help:"the probability of a single node going down within the next checker iteration" default:"0.00005435" `
	RepairQueueInsertBatchSize int      `help:"Number of damaged segments to buffer in-memory before flushing to the repair queue" default:"100" `
	RepairExcludedCountryCodes []string `help:"list of country codes to treat node from this country as offline " default:"" hidden:"true"`
	DoDeclumping               bool     `help:"Treat pieces on the same network as in need of repair" default:"true"`
	DoPlacementCheck           bool     `help:"Treat pieces out of segment placement as in need of repair" default:"true"`
}

Config contains configurable values for checker.

type Observer added in v1.79.1

type Observer struct {
	TotalStats aggregateStatsPlacements
	// contains filtered or unexported fields
}

Observer implements the ranged loop Observer interface.

architecture: Observer

func NewObserver added in v1.79.1

func NewObserver(logger *zap.Logger, repairQueue queue.RepairQueue, overlay *overlay.Service, placements nodeselection.PlacementDefinitions, config Config) *Observer

NewObserver creates new checker observer instance.

func (*Observer) Finish added in v1.79.1

func (observer *Observer) Finish(ctx context.Context) (err error)

Finish is called after all segments are processed by all observers.

func (*Observer) Fork added in v1.79.1

func (observer *Observer) Fork(ctx context.Context) (_ rangedloop.Partial, err error)

Fork creates a Partial to process a chunk of all the segments.

func (*Observer) Join added in v1.79.1

func (observer *Observer) Join(ctx context.Context, partial rangedloop.Partial) (err error)

Join is called after the chunk for Partial is done. This gives the opportunity to merge the output like in a reduce step.

func (*Observer) RefreshReliabilityCache added in v1.79.1

func (observer *Observer) RefreshReliabilityCache(ctx context.Context) error

RefreshReliabilityCache forces refreshing node online status cache.

func (*Observer) Start added in v1.79.1

func (observer *Observer) Start(ctx context.Context, startTime time.Time) (err error)

Start starts parallel segments loop.

func (*Observer) TestingCompareInjuredSegmentIDs added in v1.79.1

func (observer *Observer) TestingCompareInjuredSegmentIDs(ctx context.Context, streamIDs []uuid.UUID) error

TestingCompareInjuredSegmentIDs compares stream id of injured segment.

type ReliabilityCache

type ReliabilityCache struct {
	// contains filtered or unexported fields
}

ReliabilityCache caches known nodes for the specified staleness duration and updates automatically from overlay.

architecture: Service

func NewReliabilityCache

func NewReliabilityCache(overlay *overlay.Service, staleness time.Duration) *ReliabilityCache

NewReliabilityCache creates a new reliability checking cache.

func (*ReliabilityCache) GetNodes added in v1.89.2

func (cache *ReliabilityCache) GetNodes(ctx context.Context, validUpTo time.Time, nodeIDs []storj.NodeID, selectedNodes []nodeselection.SelectedNode) ([]nodeselection.SelectedNode, error)

GetNodes gets the cached SelectedNode records (valid as of the given time) for each of the requested node IDs, and returns them in order. If a node is not in the reliability cache (that is, it is unknown or disqualified), an empty SelectedNode will be returned for the index corresponding to that node ID. Slice selectedNodes will be filled with results nodes and returned. It's length must be equal to nodeIDs slice.

func (*ReliabilityCache) LastUpdate

func (cache *ReliabilityCache) LastUpdate() time.Time

LastUpdate returns when the cache was last updated, or the zero value (time.Time{}) if it has never yet been updated. LastUpdate() does not trigger an update itself.

func (*ReliabilityCache) NumNodes added in v1.20.1

func (cache *ReliabilityCache) NumNodes(ctx context.Context) (numNodes int, err error)

NumNodes returns the number of online active nodes (as determined by the reliability cache). This number is not guaranteed to be consistent with either the nodes database or the reliability cache after returning; it is just a best-effort count and should be treated as an estimate.

func (*ReliabilityCache) Refresh

func (cache *ReliabilityCache) Refresh(ctx context.Context) (err error)

Refresh refreshes the cache.

type RepairOverrides added in v1.18.1

type RepairOverrides struct {
	Values map[int]int
}

RepairOverrides is a configuration struct that contains a list of override repair values for various given RS combinations of k/o/n (min/success/total).

Can be used as a flag.

func (*RepairOverrides) GetOverrideValue added in v1.101.1

func (ros *RepairOverrides) GetOverrideValue(rs storj.RedundancyScheme) int32

GetOverrideValue returns the override value for an RS scheme if it exists, or 0 otherwise.

func (*RepairOverrides) GetOverrideValuePB added in v1.101.1

func (ros *RepairOverrides) GetOverrideValuePB(rs *pb.RedundancyScheme) int32

GetOverrideValuePB returns the override value for a pb RS scheme if it exists, or 0 otherwise.

func (*RepairOverrides) Set added in v1.18.1

func (ros *RepairOverrides) Set(s string) error

Set sets the value from a string in the format "k-override,k-override,...".

func (*RepairOverrides) String added in v1.18.1

func (ros *RepairOverrides) String() string

String is required for pflag.Value. It is a comma separated list of RepairOverride configs.

func (RepairOverrides) Type added in v1.18.1

func (RepairOverrides) Type() string

Type implements pflag.Value.

type RepairTargetOverrides added in v1.101.3

type RepairTargetOverrides struct {
	RepairOverrides
}

RepairTargetOverrides override values for repair success target.

type RepairThresholdOverrides added in v1.101.3

type RepairThresholdOverrides struct {
	RepairOverrides
}

RepairThresholdOverrides override values for repair threshold.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL