g2cache

package module
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Dec 22, 2023 License: Apache-2.0 Imports: 15 Imported by: 2

README

g2cache

g2cache

介绍

分布式多级缓存方案g2cache

软件架构

软件架构说明
主要内容:

模块 功能 特点 注意
local interface 本机内存高速缓存 纳秒,毫秒级别响应速度;有效期较短 实现LocalCache接口,注意控制内存
out interface 外部高速缓存,如Redis 较快的访问速度,容量很足 实现OutCache接口
LoadDataSourceFunc 外部数据源 外部加载函数,灵活,并发保护 需要自行处理panic
PubSub 外部缓存发布订阅 外部缓存可以自选实现 未实现就不支持分布式

local 本地内存高速缓存接口

// Local memory cache,Local memory cache with high access speed
type LocalCache interface {
	Get(key string, obj interface{}) (*Entry, bool, error) // obj represents the internal structure of the real object
	Set(key string, e *Entry) error                        // local storage should set Entry.Obsolete
	Del(key string) error
	ThreadSafe() // Need to ensure thread safety
	Close()
}

out 外部缓存接口,值得一提的是外部缓存需要实现发布订阅接口,这是因为g2cache支持分布式多级缓存

// External cache has faster access speed, such as Redis
type OutCache interface {
	Get(key string, obj interface{}) (*Entry, bool, error) // obj represents the internal structure of the real object
	Set(key string, e *Entry) error                        // out storage should set Entry.Expiration
	Subscribe(data chan *ChannelMeta) error
	Publish(gid ,key string, action int8, data *Entry) error
	Del(key string) error
	ThreadSafe() // Need to ensure thread safety
	Close()
}

外部缓存发布订阅

// only out storage pub sub
type PubSub interface {
	Subscribe(data chan *ChannelMeta) error
	Publish(gid, key string, action int8, data *Entry) error
}

LoadDataSourceFunc 原始数据加载函数,请自行处理panic并以error形式返回
加载函数支持返回string,map,slice,struct,ptr类型

// Shouldn't throw a panic, please return an error
type LoadDataSourceFunc func() (interface{}, error)

缓存效果-初始

[G2CACHE] 2021/04/21 11:46:21 [debug] key:[[g2cache-key:61               ]] => [ hit data source ]
[G2CACHE] 2021/04/21 11:46:21 [debug] key:[[g2cache-key:222               ]] => [ hit data source ]
[G2CACHE] 2021/04/21 11:46:21 [debug] key:[[g2cache-key:61                ]] => [ hit local storage ]
[G2CACHE] 2021/04/21 11:46:21 [debug] key:[[g2cache-key:183                ]] => [ hit data source ]
[G2CACHE] 2021/04/21 11:46:22 [debug] key:[[g2cache-key:83                ]] => [ hit data source ]
[G2CACHE] 2021/04/21 11:46:22 [debug] key:[[g2cache-key:18                ]] => [ hit data source ]
[G2CACHE] 2021/04/21 11:46:22 [debug] key:[[g2cache-key:103               ]] => [ hit local storage ]
[G2CACHE] 2021/04/21 11:46:22 [debug] key:[[g2cache-key:90                ]] => [ hit data source ]
[G2CACHE] 2021/04/21 11:46:22 [debug] key:[[g2cache-key:170               ]] => [ hit data source ]
[G2CACHE] 2021/04/21 11:46:22 [debug] key:[[g2cache-key:101               ]] => [ hit data source ]
[G2CACHE] 2021/04/21 11:46:22 [debug] key:[[g2cache-key:165               ]] => [ hit data source ]
[G2CACHE] 2021/04/21 11:46:23 [debug] statistics [local] hit percentage [[5.8824]]
[G2CACHE] 2021/04/21 11:46:23 [debug] statistics [out] hit percentage [[1.9608]]
[G2CACHE] 2021/04/21 11:46:23 [debug] statistics [data source] hit percentage [[90.1961]]

缓存效果-N分钟后

[G2CACHE] 2021/04/21 12:26:53 [debug] key:[[g2cache-key:106               ]] => [ hit out storage ]
[G2CACHE] 2021/04/21 12:26:53 [debug] key:[[g2cache-key:136               ]] => [ hit data source ]
[G2CACHE] 2021/04/21 12:26:53 [debug] key:[[g2cache-key:100               ]] => [ hit local storage ]
[G2CACHE] 2021/04/21 12:26:53 [debug] key:[[g2cache-key:219               ]] => [ hit data source ]
[G2CACHE] 2021/04/21 12:26:53 [debug] key:[[g2cache-key:13                ]] => [ hit local storage ]
[G2CACHE] 2021/04/21 12:26:53 [debug] key:[[g2cache-key:200               ]] => [ hit local storage ]
[G2CACHE] 2021/04/21 12:26:53 [debug] key:[[g2cache-key:172               ]] => [ hit data source ]
[G2CACHE] 2021/04/21 12:26:53 [debug] statistics [local] hit percentage [[45.3865]]
[G2CACHE] 2021/04/21 12:26:53 [debug] statistics [out] hit percentage [[6.7332]]
[G2CACHE] 2021/04/21 12:26:53 [debug] statistics [data source] hit percentage [[48.1297]]

缓存效果-N+M分钟后

[G2CACHE] 2021/04/21 13:18:53 [debug] key:[[g2cache-key:2                 ]] => [ hit local storage ]
[G2CACHE] 2021/04/21 13:18:53 [debug] key:[[g2cache-key:2                 ]] => [ hit out storage ]
[G2CACHE] 2021/04/21 13:18:53 [debug] key:[[g2cache-key:113               ]] => [ hit local storage ]
[G2CACHE] 2021/04/21 13:18:53 [debug] key:[[g2cache-key:113               ]] => [ hit out storage ]
[G2CACHE] 2021/04/21 13:18:53 [debug] key:[[g2cache-key:43                ]] => [ hit local storage ]
[G2CACHE] 2021/04/21 13:18:53 [debug] key:[[g2cache-key:43                ]] => [ hit out storage ]
[G2CACHE] 2021/04/21 13:18:53 [debug] statistics [local] hit percentage [[82.3877]]
[G2CACHE] 2021/04/21 13:18:53 [debug] statistics [out] hit percentage [[16.2689]]
[G2CACHE] 2021/04/21 13:18:53 [debug] statistics [data source] hit percentage [[1.3641]]
Usage

go get -u gitee.com/kelvins-io/[email protected]

模拟测试

修改example/main.go中Redis的配置
sh example-run.sh

分支说明
  1. copyobj分支支持返回特定object,为不稳定版本
  2. dev分支Get返回object序列化值,过渡版
  3. master分支只提供sync.Map的local实现,早期版
  4. release分支提供发布版,与copyobj有较大变化
参与贡献
  1. Fork 本仓库
  2. 新建 Feat_xxx 分支
  3. 提交代码
  4. 新建 Pull Request
合作交流

QQ群:
g2cache

邮箱: [email protected]

Documentation

Index

Constants

View Source
const (
	SetPublishType int8 = iota
	DelPublishType
)

Entry expire is UnixNano

Variables

View Source
var (
	CacheDebug                  bool
	CacheMonitor                bool
	OutCachePubSub              bool
	CacheMonitorSecond          = 5
	DefaultGPoolWorkerNum       = 200
	DefaultGPoolJobQueueChanLen = 1000
)
View Source
var (
	CacheKeyEmpty           = errors.New("cache key is empty")
	CacheObjNil             = errors.New("cache object is nil")
	LoadDataSourceFuncNil   = errors.New("cache load func is nil")
	LocalStorageClose       = errors.New("local storage close !!! ")
	OutStorageClose         = errors.New("out storage close !!! ")
	CacheClose              = errors.New("g2cache close !!! ")
	DataSourceLoadNil       = errors.New("data source load nil")
	OutStorageLoadNil       = errors.New("out storage load nil")
	CacheNotImplementPubSub = errors.New("cache not implement pubsub interface")
)
View Source
var (
	DefaultFreeCacheSize = 50 * 1024 * 1024 // 50MB
)
View Source
var DefaultPubSubRedisChannel = "g2cache-pubsub-channel"
View Source
var (
	EntryLazyFactor = 32
)

Functions

func GenKey

func GenKey(args ...interface{}) string

func GetRedisPool

func GetRedisPool(conf *RedisConf) (*redis.Pool, error)

func LogDebug

func LogDebug(s ...interface{})

func LogDebugF

func LogDebugF(f string, s ...interface{})

func LogErr

func LogErr(s ...interface{})

func LogErrF

func LogErrF(f string, s ...interface{})

func LogInfo

func LogInfo(s ...interface{})

func LogInfoF

func LogInfoF(f string, s ...interface{})

func NewUUID

func NewUUID() (string, error)

func RedisDelKey

func RedisDelKey(key string, pool *redis.Pool) error

func RedisGetString

func RedisGetString(key string, pool *redis.Pool) (string, error)

func RedisPublish

func RedisPublish(channel, message string, pool *redis.Pool) error

func RedisSetString

func RedisSetString(key, value string, ttl int, pool *redis.Pool) error

Types

type ChannelMeta

type ChannelMeta struct {
	Key    string `json:"key"`    // cache key
	Gid    string `json:"gid"`    // Used to identify working groups
	Action int8   `json:"action"` // SetPublishType,DelPublishType
	Data   *Entry `json:"data"`
}

type Entry

type Entry struct {
	Value      interface{} `json:"value"`
	TtlSecond  int         `json:"ttl"`
	Obsolete   int64       `json:"obsolete"`
	Expiration int64       `json:"expiration"`
}

func NewEntry

func NewEntry(v interface{}, second int) *Entry

func (*Entry) Expired

func (e *Entry) Expired() bool

Expired means that the data is unavailable and data needs to be synchronized

func (*Entry) GetExpireTTL

func (e *Entry) GetExpireTTL() (second int64)

func (*Entry) GetObsoleteTTL

func (e *Entry) GetObsoleteTTL() (second int64)

func (*Entry) Obsoleted

func (e *Entry) Obsoleted() bool

Outdated data means that the data is still available, but not up-to-date

func (*Entry) String

func (e *Entry) String() string

type FreeCache

type FreeCache struct {
	// contains filtered or unexported fields
}

func NewFreeCache

func NewFreeCache() *FreeCache

func (*FreeCache) Close

func (c *FreeCache) Close()

func (*FreeCache) Del

func (c *FreeCache) Del(key string) error

func (*FreeCache) Get

func (c *FreeCache) Get(key string, obj interface{}) (*Entry, bool, error)

func (*FreeCache) Set

func (c *FreeCache) Set(key string, e *Entry) error

func (*FreeCache) ThreadSafe

func (c *FreeCache) ThreadSafe()

type G2Cache

type G2Cache struct {
	GID string // Identifies the number of an instance
	// contains filtered or unexported fields
}

func New

func New(out OutCache, local LocalCache) (g *G2Cache, err error)

func (*G2Cache) Close

func (g *G2Cache) Close()

func (*G2Cache) Del

func (g *G2Cache) Del(key string, wait bool) (err error)

func (*G2Cache) Get

func (g *G2Cache) Get(key string, ttlSecond int, obj interface{}, fn LoadDataSourceFunc) error

Get return err DataSourceLoadNil if fn exec return nil

func (*G2Cache) Set

func (g *G2Cache) Set(key string, obj interface{}, ttlSecond int, wait bool) (err error)

type Harsher

type Harsher interface {
	Sum64(string) uint64
}

type HitStatistics

type HitStatistics struct {
	HitOutStorageTotalRate   float64 `json:"hit_out_storage_total_rate"`
	HitDataSourceTotalRate   float64 `json:"hit_data_source_total_rate"`
	HitLocalStorageTotalRate float64 `json:"hit_local_storage_total_rate"`
	HitDataSourceTotal       int64   `json:"hit_data_source_total"`
	HitLocalStorageTotal     int64   `json:"hit_local_storage_total"`
	HitOutStorageTotal       int64   `json:"hit_out_storage_total"`
	AccessGetTotal           int64   `json:"access_get_total"`
}
var HitStatisticsOut HitStatistics

func (*HitStatistics) Calculation

func (h *HitStatistics) Calculation()

func (*HitStatistics) StatisticsDataSource

func (h *HitStatistics) StatisticsDataSource()

func (*HitStatistics) StatisticsLocalStorage

func (h *HitStatistics) StatisticsLocalStorage()

func (*HitStatistics) StatisticsOutStorage

func (h *HitStatistics) StatisticsOutStorage()

func (*HitStatistics) String

func (h *HitStatistics) String() string

type Job

type Job func()

Represents user request, function which should be executed in some worker.

type LoadDataSourceFunc

type LoadDataSourceFunc func() (interface{}, error)

Shouldn't throw a panic, please return an error

type LocalCache

type LocalCache interface {
	Get(key string, obj interface{}) (*Entry, bool, error) // obj represents the internal structure of the real object
	Set(key string, e *Entry) error                        // local storage should set Entry.Obsolete
	Del(key string) error
	ThreadSafe() // Need to ensure thread safety
	Close()
}

Local memory cache,Local memory cache with high access speed

type LoggerInterface

type LoggerInterface interface {
	LogInfoF(f string, s ...interface{})
	LogInfo(s ...interface{})
	LogDebug(s ...interface{})
	LogDebugF(f string, s ...interface{})
	LogErr(s ...interface{})
	LogErrF(f string, s ...interface{})
}

外部调用者可实现此日志接口用于将日志导出

var (
	Logger LoggerInterface = &sysLogger{}
)

type OutCache

type OutCache interface {
	Get(key string, obj interface{}) (*Entry, bool, error) // obj represents the internal structure of the real object
	Set(key string, e *Entry) error                        // out storage should set Entry.Expiration
	Del(key string) error
	ThreadSafe() // Need to ensure thread safety
	Close()
}

External cache has faster access speed, such as Redis

type Pool

type Pool struct {
	// contains filtered or unexported fields
}

func NewPool

func NewPool(numWorkers int, jobQueueLen int) *Pool

Will make pool of gorouting workers. numWorkers - how many workers will be created for this pool queueLen - how many jobs can we accept until we block

Returned object contains JobQueue reference, which you can use to send job to pool.

func (*Pool) Release

func (p *Pool) Release()

Will release resources used by pool

func (*Pool) SendJob

func (p *Pool) SendJob(job func())

func (*Pool) SendJobWithDeadline

func (p *Pool) SendJobWithDeadline(job func(), t time.Time) bool

func (*Pool) SendJobWithTimeout

func (p *Pool) SendJobWithTimeout(job func(), t time.Duration) bool

type PubSub

type PubSub interface {
	Subscribe(data chan<- *ChannelMeta) error
	Publish(gid, key string, action int8, data *Entry) error
}

only out storage pub sub

type RedisCache

type RedisCache struct {
	// contains filtered or unexported fields
}

func NewRedisCache

func NewRedisCache() (*RedisCache, error)

func (*RedisCache) Close

func (r *RedisCache) Close()

func (*RedisCache) Del

func (r *RedisCache) Del(key string) error

func (*RedisCache) DistributedEnable

func (r *RedisCache) DistributedEnable() bool

func (*RedisCache) Get

func (r *RedisCache) Get(key string, obj interface{}) (*Entry, bool, error)

func (*RedisCache) Publish

func (r *RedisCache) Publish(gid, key string, action int8, value *Entry) error

func (*RedisCache) Set

func (r *RedisCache) Set(key string, obj *Entry) error

func (*RedisCache) Subscribe

func (r *RedisCache) Subscribe(ch chan<- *ChannelMeta) error

func (*RedisCache) ThreadSafe

func (r *RedisCache) ThreadSafe()

type RedisConf

type RedisConf struct {
	DSN     string
	Pwd     string
	DB      int
	MaxConn int
}
var DefaultPubSubRedisConf RedisConf
var DefaultRedisConf RedisConf

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL