Documentation ¶
Index ¶
- Constants
- func NewCopyCRDBDirect(logger zerolog.Logger, target *pgx.Conn) *copyCRDBDirect
- func NewGCPStore(logger zerolog.Logger, client *storage.Client, creds *google.Credentials, ...) *gcpStore
- func NewLocalStore(logger zerolog.Logger, basePath string, listenAddr string, ...) (*localStore, error)
- func NewS3Store(logger zerolog.Logger, session *session.Session, creds credentials.Value, ...) *s3Store
- func SetCopyEnvVars(ctx context.Context, conn *pgx.Conn) error
- type DatastoreCreationPayload
- type DirectCopyPayload
- type GCPPayload
- type GCPStorageWriterMock
- type LocalPathPayload
- type Resource
- type S3Payload
- type Store
Constants ¶
View Source
const AWSUploadFileMockErrMsg = "mocked error for uploading file for aws"
View Source
const DirectCopyWriterMockErrMsg = "forced error for direct copy"
View Source
const GCPWriterMockErrMsg = "forced error for gcp storage writer"
View Source
const LocalWriterMockErrMsg = "forced error for local path storage"
Variables ¶
This section is empty.
Functions ¶
func NewCopyCRDBDirect ¶
func NewGCPStore ¶
func NewLocalStore ¶
func NewS3Store ¶
func SetCopyEnvVars ¶ added in v0.1.3
Types ¶
type DatastoreCreationPayload ¶ added in v0.1.1
type DatastoreCreationPayload struct { DirectCopyPl *DirectCopyPayload GCPPl *GCPPayload S3Pl *S3Payload LocalPathPl *LocalPathPayload TestFailedWriteToBucket bool }
type DirectCopyPayload ¶ added in v0.1.1
type DirectCopyPayload struct {
TargetConnForCopy *pgx.Conn
}
type GCPPayload ¶ added in v0.1.1
type GCPStorageWriterMock ¶ added in v0.1.1
GCPStorageWriterMock is to mock a gcp storage that always fail to upload to the bucket. We use it to simulate a disastrous edge case and ensure that the error in this case would be properly propagated.
type LocalPathPayload ¶ added in v0.1.1
type Store ¶
type Store interface { // CreateFromReader is responsible for the creation of the individual // CSVs from the data export process. It will create the file and upload // it to the respetive data store and return the resource object which // will be used in the data import phase. CreateFromReader(ctx context.Context, r io.Reader, table dbtable.VerifiedTable, iteration int, fileExt string, numRows chan int, testingKnobs testutils.FetchTestingKnobs, shardNum int) (Resource, error) // ListFromContinuationPoint is used when restarting Fetch from // a continuation point. It will query the respective data store // and create the slice of resources that will be used by the // import process. Note that NO files are created from the method. // It simply lists all files in the data store and filters and returns // the files that are needed. ListFromContinuationPoint(ctx context.Context, table dbtable.VerifiedTable, fileName string) ([]Resource, error) CanBeTarget() bool DefaultFlushBatchSize() int Cleanup(ctx context.Context) error TelemetryName() string }
Click to show internal directories.
Click to hide internal directories.