neptune

package
v2.17.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 12, 2023 License: MIT Imports: 20 Imported by: 0

Documentation

Overview

This module, when combined with codelist.go, provides code that satisfies the graph.driver.CodeList interface using Gremlin queries into a Neptune database.

It is dedicated to code to satisfy the GetCodeDatasets() method - which is sufficiently complex to merit a module (and tests) of its own.

Index

Constants

This section is empty.

Variables

View Source 
var ErrInvalidFilter = errors.New("nil filter cannot be processed")

ErrInvalidFilter is returned if the provided filter is nil.

Functions

This section is empty.

Types

type NeptuneDB 

type NeptuneDB struct {
	neptune.NeptuneDriver
	// contains filtered or unexported fields
}

func New 

func New(dbAddr string, size, timeout, retries, batchSizeReader, batchSizeWriter, maxWorkers int, tlsSkip bool, retryTime time.Duration, errs chan error) (n *NeptuneDB, err error)

func (*NeptuneDB) AddDimensions 

func (n *NeptuneDB) AddDimensions(ctx context.Context, instanceID string, dimensions []interface{}) error

AddDimensions list to the specified instance node

func (*NeptuneDB) AddVersionDetailsToInstance 

func (n *NeptuneDB) AddVersionDetailsToInstance(ctx context.Context, instanceID string, datasetID string, edition string, version int) error

AddVersionDetailsToInstance updates an instance node to contain details of which dataset, edition and version the instance will also be known by

func (*NeptuneDB) CloneNodes 

func (n *NeptuneDB) CloneNodes(ctx context.Context, attempt int, instanceID, codeListID, dimensionName string) (err error)

func (*NeptuneDB) CloneNodesFromIDs added in v2.5.0 

func (n *NeptuneDB) CloneNodesFromIDs(ctx context.Context, attempt int, instanceID, codeListID, dimensionName string, ids map[string]string, hasData bool) (err error)

CloneNodesFromIDs clones the generic hierarchy nodes with the provided IDs (map for uniqueness and efficiency)

func (*NeptuneDB) CloneOrderFromIDs added in v2.10.0 

func (n *NeptuneDB) CloneOrderFromIDs(ctx context.Context, codeListID string, ids map[string]string) (err error)

CloneOrderFromIDs copies the order property from the 'usedBy' edge that goes from the code node to the provided codelist node where the code node is the determined by the 'hasCode' edge of the generic hierarchy nodes. The order property is stored as a property of the clone node (assumes a clone_of edge exists from a hierarchy node to the generic hierarchy node)

func (*NeptuneDB) CloneRelationships 

func (n *NeptuneDB) CloneRelationships(ctx context.Context, attempt int, instanceID, codeListID, dimensionName string) (err error)

func (*NeptuneDB) CloneRelationshipsFromIDs added in v2.5.0 

func (n *NeptuneDB) CloneRelationshipsFromIDs(ctx context.Context, attempt int, instanceID, dimensionName string, ids map[string]string) error

CloneRelationshipsFromIDs clones the has_parent edges between clones that have parent relationship according to the provided generic hierarchy nodes.

func (*NeptuneDB) CountCodes added in v2.7.2 

func (n *NeptuneDB) CountCodes(ctx context.Context, codeListID, edition string) (int64, error)

CountCodes counts the number of codes corresponding to the provided codeListID and edition.

func (*NeptuneDB) CountInsertedObservations 

func (n *NeptuneDB) CountInsertedObservations(ctx context.Context, instanceID string) (count int64, err error)

CountInsertedObservations returns the current number of observations relating to a given instance

func (*NeptuneDB) CountNodes 

func (n *NeptuneDB) CountNodes(ctx context.Context, instanceID, dimensionName string) (count int64, err error)

CountNodes returns the number of hierarchy nodes for the provided instanceID and dimensionName

func (*NeptuneDB) CreateCodeRelationship 

func (n *NeptuneDB) CreateCodeRelationship(ctx context.Context, instanceID, codeListID, code string) error

CreateCodeRelationship links an instance to a code for the given dimension option

func (*NeptuneDB) CreateHasCodeEdges added in v2.10.0 

func (n *NeptuneDB) CreateHasCodeEdges(ctx context.Context, attempt int, codeListID string, codesById map[string]string) (err error)

func (*NeptuneDB) CreateInstance 

func (n *NeptuneDB) CreateInstance(ctx context.Context, instanceID string, csvHeaders []string) error

CreateInstance will check if an instance node already exists and create one from the provided details if one does not exist

func (*NeptuneDB) CreateInstanceConstraint 

func (n *NeptuneDB) CreateInstanceConstraint(ctx context.Context, instanceID string) error

CreateInstanceConstraint is not needed for the neptune implementation, as constraints are not a neptune construct

func (*NeptuneDB) CreateInstanceHierarchyConstraints 

func (n *NeptuneDB) CreateInstanceHierarchyConstraints(ctx context.Context, attempt int, instanceID, dimensionName string) error

func (*NeptuneDB) GetCode 

func (n *NeptuneDB) GetCode(ctx context.Context, codeListID, edition string, code string) (*models.Code, error)

GetCode provides a Code struct to represent the requested code list, edition and code string. E.g. ashe-earnings|one-off|hourly-pay-gross. It doesn't need to access the database to form the response, but does so to validate the query. Specifically it can return errors as follows: - The Gremlin query failed to execute. - The query parameter values do not successfully navigate to a Code node. (error is `ErrNotFound`) - Duplicate Code(s) exist that satisfy the search criteria (error is `ErrMultipleFound`)

func (*NeptuneDB) GetCodeDatasets 

func (n *NeptuneDB) GetCodeDatasets(ctx context.Context, codeListID, edition string, code string) (*models.Datasets, error)

GetCodeDatasets searches the database for datasets that are associated with the given code list, code, and code list edition. Specifically those that satisfy all of:

  1. code lists that match the requested code list ID.
  2. code lists of the requested edition.
  3. codes that match the requested code value.
  4. datasets that are related to qualifying codes by *inDataset* edges.
  5. datasets that have the *isPublished* state true.

Each such result from the database (potentially) has the properties:

  • dimensionName (what the dataset calls this dimension)
  • datasetEdition
  • version

The results however include all permutations of dimensionName and datasetEdition - BUT ONLY CITES the most recent dataset *version* of those found for that permutation.

func (*NeptuneDB) GetCodeList 

func (n *NeptuneDB) GetCodeList(ctx context.Context, codeListID string) (
	*models.CodeList, error)

GetCodeList provides a CodeList for a given ID (e.g. "ashe-earnings"), having checked it exists in the database. Nb. The caller is expected to fully qualify the embedded Links field afterwards. It returns an error if: - The Gremlin query failed to execute. - The requested CodeList does not exist. (error is `ErrNotFound`) - Duplicate CodeLists exist with the given ID (error is `ErrMultipleFound`)

func (*NeptuneDB) GetCodeLists 

func (n *NeptuneDB) GetCodeLists(ctx context.Context, filterBy string) (*models.CodeListResults, error)

GetCodeLists provides a list of either all Code Lists, or a list of only those having a boolean property with the name <filterBy> which is set to true. E.g. "geography": true. The caller is expected to fully qualify the embedded Links field afterwards. It returns an error if: - The Gremlin query failed to execute. - A CodeList is encountered that does not have *listID* property.

func (*NeptuneDB) GetCodes 

func (n *NeptuneDB) GetCodes(ctx context.Context, codeListID, edition string) (*models.CodeResults, error)

GetCodes provides a list of Code(s) packaged into a models.CodeResults structure that has been populated by a database query that finds the Code List nodes of the required codeListID (e.g. "ashe-earnings"), and the required edition (e.g. "one-off"), and then harvests the Code nodes that are known to be "usedBy" that Code List. It raises a wrapped error if the database raises a non-transient error, (e.g. malformed query). It raises driver.ErrNotFound if the graph traversal above produces an empty list of codes - including the case of a short-circuit early termination of the query, because no such qualifying code list exists. It returns a wrapped error if a Code is found that does not have a "value" property.

func (*NeptuneDB) GetCodesOrder added in v2.9.0 

func (n *NeptuneDB) GetCodesOrder(ctx context.Context, codeListID string, codes []string) (codeOrders map[string]*int, err error)

GetCodesOrder obtains the numerical order value defined in the 'usedBy' edge between the provided codes and codeListID nodes

func (*NeptuneDB) GetCodesWithData added in v2.5.0 

func (n *NeptuneDB) GetCodesWithData(ctx context.Context, attempt int, instanceID, dimensionName string) (codes []string, err error)

GetCodesWithData returns a list of values that are present in nodes with label _{instanceID}_{dimensionName}

func (*NeptuneDB) GetEdition 

func (n *NeptuneDB) GetEdition(ctx context.Context, codeListID, edition string) (*models.Edition, error)

GetEdition provides an Edition structure for the code list in the database that has both the given codeListID (e.g. "ashed-earnings"), and the given edition string (e.g. "one-off"). Nb. The caller is expected to fully qualify the embedded Links field afterwards. It returns an error if: - The Gremlin query failed to execute. (wrapped error) - No CodeLists exist with the requested codeListID (error is `ErrNotFound`) - A CodeList is found that does not have the "edition" property (error is 'ErrNoSuchProperty') - More than one CodeList exists with the requested ID AND edition (error is `ErrMultipleFound`)

func (*NeptuneDB) GetEditions 

func (n *NeptuneDB) GetEditions(ctx context.Context, codeListID string) (*models.Editions, error)

GetEditions provides a models.Editions structure populated based on the the values in the Code List vertices in the database, that have the provided codeListId. It returns an error if: - The Gremlin query failed to execute. (wrapped error) - No CodeLists are found of the requested codeListID (error is ErrNotFound') - A CodeList is found that does not have the "edition" property (error is 'ErrNoSuchProperty')

func (*NeptuneDB) GetGenericHierarchyAncestriesIDs added in v2.5.0 

func (n *NeptuneDB) GetGenericHierarchyAncestriesIDs(ctx context.Context, attempt int, codeListID string, codes []string) (nodeIDs map[string]string, err error)

GetGenericHierarchyAncestriesIDs obtains a list of node IDs for the parents of the hierarchy nodes that have a code in the provided list.

func (*NeptuneDB) GetGenericHierarchyNodeIDs added in v2.5.0 

func (n *NeptuneDB) GetGenericHierarchyNodeIDs(ctx context.Context, attempt int, codeListID string, codes []string) (nodeIDs map[string]string, err error)

GetGenericHierarchyNodeIDs obtains a list of node IDs for generic hierarchy nodes for the provided codeListID, which have a code in the provided list.

func (*NeptuneDB) GetHierarchyCodelist 

func (n *NeptuneDB) GetHierarchyCodelist(ctx context.Context, instanceID, dimension string) (codelistID string, err error)

func (*NeptuneDB) GetHierarchyElement 

func (n *NeptuneDB) GetHierarchyElement(ctx context.Context, instanceID, dimension, code string) (node *models.HierarchyResponse, err error)

func (*NeptuneDB) GetHierarchyNodeIDs added in v2.5.0 

func (n *NeptuneDB) GetHierarchyNodeIDs(ctx context.Context, attempt int, instanceID, dimensionName string) (ids map[string]string, err error)

GetHierarchyNodeIDs returns a map of IDs for the cloned hierarchy nodes for a provided instanceID and dimensionName

func (*NeptuneDB) GetHierarchyRoot 

func (n *NeptuneDB) GetHierarchyRoot(ctx context.Context, instanceID, dimension string) (node *models.HierarchyResponse, err error)

func (*NeptuneDB) HierarchyExists added in v2.8.0 

func (n *NeptuneDB) HierarchyExists(ctx context.Context, instanceID, dimension string) (hierarchyExists bool, err error)

func (*NeptuneDB) InsertDimension 

func (n *NeptuneDB) InsertDimension(ctx context.Context, uniqueDimensions map[string]string, uniqueDimensionsMutex *sync.Mutex, instanceID string, d *models.Dimension) (*models.Dimension, error)

InsertDimension node to neptune and create relationships to the instance node. Where nodes and relationships already exist, ensure they are upserted.

func (*NeptuneDB) InsertObservationBatch 

func (n *NeptuneDB) InsertObservationBatch(ctx context.Context, attempt int, instanceID string, observations []*models.Observation, dimensionNodeIDs map[string]string) error

InsertObservationBatch creates a batch query based on a provided list of observations and attempts to insert them by bulk to the database

func (*NeptuneDB) InstanceExists 

func (n *NeptuneDB) InstanceExists(ctx context.Context, instanceID string) (bool, error)

InstanceExists returns true if an instance already exists with the provided id

func (*NeptuneDB) MarkNodesToRemain 

func (n *NeptuneDB) MarkNodesToRemain(ctx context.Context, attempt int, instanceID, dimensionName string) (err error)

func (*NeptuneDB) RemoveCloneEdges 

func (n *NeptuneDB) RemoveCloneEdges(ctx context.Context, attempt int, instanceID, dimensionName string) (err error)

func (*NeptuneDB) RemoveCloneEdgesFromSourceIDs added in v2.5.0 

func (n *NeptuneDB) RemoveCloneEdgesFromSourceIDs(ctx context.Context, attempt int, ids map[string]string) (err error)

RemoveCloneEdgesFromSourceIDs removes the 'clone-of' edges between a set of cloned nodes and their corresponding generic hierarchy nodes.

func (*NeptuneDB) RemoveNodesNotMarkedToRemain 

func (n *NeptuneDB) RemoveNodesNotMarkedToRemain(ctx context.Context, attempt int, instanceID, dimensionName string) (err error)

func (*NeptuneDB) RemoveRemainMarker 

func (n *NeptuneDB) RemoveRemainMarker(ctx context.Context, attempt int, instanceID, dimensionName string) (err error)

func (*NeptuneDB) SetHasData 

func (n *NeptuneDB) SetHasData(ctx context.Context, attempt int, instanceID, dimensionName string) (err error)

func (*NeptuneDB) SetInstanceIsPublished 

func (n *NeptuneDB) SetInstanceIsPublished(ctx context.Context, instanceID string) error

SetInstanceIsPublished sets a flag on an instance node to indicate the published state

func (*NeptuneDB) SetNumberOfChildren 

func (n *NeptuneDB) SetNumberOfChildren(ctx context.Context, attempt int, instanceID, dimensionName string) (err error)

func (*NeptuneDB) SetNumberOfChildrenFromIDs added in v2.5.0 

func (n *NeptuneDB) SetNumberOfChildrenFromIDs(ctx context.Context, attempt int, ids map[string]string) (err error)

SetNumberOfChildrenFromIDs sets a property called 'numberOfChildren' to the value indegree of edges 'hasParent' for the provided node IDs

func (*NeptuneDB) StreamCSVRows 

func (n *NeptuneDB) StreamCSVRows(ctx context.Context, instanceID, filterID string, filter *observation.DimensionFilters, limit *int) (observation.StreamRowReader, error)

StreamCSVRows returns a reader allowing individual CSV rows to be read. Rows returned can be limited, to stop this pass in nil. If filter.DimensionFilters is nil, empty or contains only empty values then a StreamRowReader for the entire dataset will be returned.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.