pinecone

package
v0.0.0-...-1d0f068 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 1, 2024 License: Apache-2.0 Imports: 14 Imported by: 0

Documentation

Overview

Package pinecone implements a genkit plugin for the Pinecone vector database. This defines an indexer and a retriever.

Accessing Pinecone requires an API key, which is passed as an argument to the relevant functions. Passing the API key as the empty string directs this package to use the PINECONE_API_KEY environment variable.

All Pinecone data is stored in what Pinecone calls an index. An index may be serverless, meaning that it scales based on usage, or it may be pod-based, meaning that it uses a specified hardware configuration (which Pinecone calls a pod). Each index has an associated host URL. Storing and retrieving data requires passing this URL. The URL may be retrieved using the Indexes or IndexData function; it's the Host field in the returned struct.

Indexes can be partitioned into namespaces. Operations that use indexes pass a namespace argument, where the empty string represents the default namespace.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func New 

func New(ctx context.Context, apiKey, host string, embedder ai.Embedder, embedderOptions any, textKey string) (ai.DocumentStore, error)

New returns an ai.DocumentStore that uses Pinecone.

apiKey is the API key to use to access Pinecone. If it is the empty string, it is read from the PINECONE_API_KEY environment variable.

host is the controller host name. This implies which index to use. If it is the empty string, it is read from the PINECONE_CONTROLLER_HOST environment variable.

The caller must pass an Embedder to use.

The textKey parameter is the metadata key to use to store document text in Pinecone; the default is "_content".

Types

type Client 

type Client struct {
	// contains filtered or unexported fields
}

A Client is used to perform database operations.

func NewClient 

func NewClient(ctx context.Context, apiKey string) (*Client, error)

NewClient builds a Client.

apiKey is the API key to use to access Pinecone. If it is the empty string, it is read from the PINECONE_API_KEY environment variable.

func (*Client) Index 

func (c *Client) Index(ctx context.Context, host string) (*Index, error)

The Index method returns an Index, used to access a specific Pinecone index.

host is the controller host name. This implies which index to use. This comes from the [IndexData.Host] field of the IndexData struct describing the desired index. If it is the empty string, it is read from the PINECONE_CONTROLLER_HOST environment variable.

func (*Client) IndexData 

func (c *Client) IndexData(ctx context.Context, index string) (*IndexData, error)

IndexData fetches the data for a specific index.

func (*Client) Indexes 

func (c *Client) Indexes(ctx context.Context) ([]IndexData, error)

Indexes fetches the available indexes.

type Index 

type Index struct {
	// contains filtered or unexported fields
}

Index is used to access a specific Pinecone index.

func (*Index) DeleteByID 

func (idx *Index) DeleteByID(ctx context.Context, ids []string, namespace string) error

Delete deletes vectors from the database by ID.

func (*Index) Query 

func (idx *Index) Query(ctx context.Context, values []float32, count int, want WantData, namespace string) ([]*QueryResult, error)

Query looks up a vector in the database. It returns a set of similar vectors. The count parameter is the maximum number of vectors to return.

func (*Index) QueryByID 

func (idx *Index) QueryByID(ctx context.Context, id string, want WantData, namespace string) (*QueryResult, error)

QueryByID looks up a vector in the database by ID.

func (*Index) Stats 

func (idx *Index) Stats(ctx context.Context) (*Stats, error)

Stats returns statistics about an index.

func (*Index) Upsert 

func (idx *Index) Upsert(ctx context.Context, vectors []Vector, namespace string) error

Upsert writes a set of vector records into the index. If a record ID already exists, the existing record is replaced with the new one. The namespace indicates which namespace to write to; an empty string means the default namespace.

The Pinecone docs say that after an Upsert operation, the vectors may not be immediately visible. The Stats method will report whether the vectors can be seen.

type IndexData 

type IndexData struct {
	Name      string `json:"name"`      // index name
	Dimension int    `json:"dimension"` // dimension of vectors in index
	Host      string `json:"host"`      // index host name
	Metric    string `json:"metric"`    // index metric: euclidean, cosine, dotproduct
	Spec      struct {
		Pod        *Pod        `json:"pod,omitempty"`       // for pod-based indexes
		Serverless *Serverless `json:serverless,omitempty"` // for serverless indexes
	} `json:"spec"`
	Status struct {
		Ready bool   `json:"ready"` // whether the index is ready
		State string `json:"state"` // ready, etc.
	} `json:"status"`
}

An IndexData contains information about a single Pinecone index.

type IndexerOptions 

type IndexerOptions struct {
	Namespace string `json:"namespace,omitempty"` // Pinecone namespace to use
}

IndexerOptions may be passed in the Options field of ai.IndexerRequest to pass options to Pinecone. The Options field should be either nil or a value of type *IndexerOptions.

type NamespaceStats 

type NamespaceStats struct {
	Count int `json:"vectorCount,omitempty"` // number of vectors in namespace
}

NamespaceStats is data returned by the Index.Stats method for a namespace.

type Pod 

type Pod struct {
	Environment      string       `json:"environment"` // index environment: hosting provider and region
	Metadata         *PodMetadata `json:metadata_config,omitempty"`
	Type             string       `json:"pod_type"`                    // pod type and size
	Pods             int          `json:"pods"`                        // number of pods in index
	Replicas         int          `json:"replicas"`                    // number of replicas
	Shards           int          `json:"shards"`                      // number of shards
	SourceCollection string       `json:"source_collection,omitempty"` // name of collection from which to create an index
}

A Pod is information about a pod-based index. Pod-based indexes are associated with a set of hardware configurations.

type PodMetadata 

type PodMetadata struct {
	Indexes []string `json:"indexes,omitempty"` // list of fields that should be indexed; nil for all
}

PodMetadata configures the metadata index.

type QueryResult 

type QueryResult struct {
	ID           string         `json:"id,omitempty"`
	Score        float32        `json:"score,omitempty"` // A higher score is more similar
	Values       []float32      `json:"values,omitempty"`
	SparseValues *SparseValues  `json:"sparse_values,omitempty"`
	Metadata     map[string]any `json:"metadata,omitempty"`
}

QueryResult is a single vector returned by a Query operation.

type RetrieverOptions 

type RetrieverOptions struct {
	Namespace string `json:"namespace,omitempty"` // Pinecone namespace to use
	Count     int    `json:"count,omitempty"`     // maximum number of values to retrieve
}

RetrieverOptions may be passed in the Options field of ai.RetrieverRequest to pass options to Pinecone. The Options field should be either nil or a value of type *RetrieverOptions.

type Serverless 

type Serverless struct {
	Cloud  string `json:"cloud"`  // Cloud name
	Region string `json:"region"` // Cloud region
}

A Serverless is information about a serverless index. Serverless indexes scale automatically with usage.

type SparseValues 

type SparseValues struct {
	Indices []uint32  `json:"indices,omitempty"`
	Values  []float32 `json:"values,omitempty"`
}

SparseValues can be used if most values in a vector are zero. Instead of listing all the values in the vector, SparseValues uses two slices of the same length, such that the real vector can be constructed with 

for i, ind := range sv.Indices {
	v[ind] = Values[i]
}

type Stats 

type Stats struct {
	Dimension  int                        `json:"dimension,omitempty"`        // index dimension
	Fullness   float32                    `json:"indexFullness,omitempty"`    // fullness, only for pod-based indexes
	Count      int                        `json:"totalVectorCount,omitempty"` // number of vectors in index
	Namespaces map[string]*NamespaceStats `json:"namespaces,omitempty"`
}

Stats is the data returns by the Index.Stats method.

type Vector 

type Vector struct {
	ID           string         `json:"id"`                      // vector ID
	Values       []float32      `json:"values,omitempty"`        // vector values
	SparseValues *SparseValues  `json:"sparse_values,omitempty"` // sparse vector values
	Metadata     map[string]any `json:"metadata,omitempty"`      // associated metadata; may be nil
}

Vector is a single vector stored in the index. Either Values or SparseValues should hold the actual data.

type WantData 

type WantData int

WantData is a set of flags that indicates which information to return when looking up a vector in the database. 

const (
	WantValues   WantData = 1 << iota // return values of matching vectors
	WantMetadata                      // return metadata of matching vectors
)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.