storage

package
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 6, 2022 License: MIT Imports: 8 Imported by: 0

Documentation

Overview

storage package has implementation of storages for small objects and index storage for big objects.

Storage for small objects is simple structure with hash maps and lists which are filled with parsed records during parsing of heap dump. It's in-memory storage and to avoid parsing small objects every time, it has serialization machinery that allows to put all contents to the file and restore it. Data is serialized to <heap dump name>.db/small-records.bin

Storage for big objects does not try to hold all in RAM. Instead it stores index to the index file that later is used to access that records. Index files are <heap dump>.db/instance-dump.idx.bin, <heap dump>.db/obj-array-dump.idx.bin and <heap dump>.db/prim-array-dump.idx.bin

For effective way of accessing index records binary search is used. Index records are key:value pairs of 8+8=16 bytes which is used to store offsets of some objects in .hprof file. All objects in .hprof files have unique identifiers and correspoiding offsets. F.e. "instance dump" object with object id = 1 and offset = 1 could be written to index file and the offset could be obtained later and the whole class dump could be parsed. Typically, using index file is more effective than linear search in .hprof file. Hovewer, it could not be true for machines with HDD disk because this mechanism assumes the data is in increasing sorted order by key because Get() is binary search. The package could be easily misused - data should be written in sorted order or should be sorted after write before read. For most sections of .hprof such as instance dumps and array dumps it's true by default - data is stored in the dump already sorted. But anyway, the order is tracked during parsing to detect anomalies in records order.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Batch 

type Batch struct {
	// contains filtered or unexported fields
}

Batch should be used to group index records for writing them more effectiveley.

type BatchSize 

type BatchSize int
const (
	DefaultBatchSize BatchSize = 1_000_000 // Means the size of 1 batch by default if 16 Mb
)

type BigRecordsReadStorage 

type BigRecordsReadStorage struct {
	// contains filtered or unexported fields
}

func NewBigRecordsReadStorage 

func NewBigRecordsReadStorage(
	instanceDumpPersistent IndexRecordsReaderAtCloser,
	instanceDumpPersistentSize int,
	objArrayDumpPersistent IndexRecordsReaderAtCloser,
	objArrayDumpPersistentSize int,
	primArrayDumpPersistent IndexRecordsReaderAtCloser,
	primArrayDumpPersistentSize int,
) (*BigRecordsReadStorage, error)

func (*BigRecordsReadStorage) Close 

func (r *BigRecordsReadStorage) Close() error

func (*BigRecordsReadStorage) HprofGcInstanceDumpGetOffset 

func (r *BigRecordsReadStorage) HprofGcInstanceDumpGetOffset(objectId core.Identifier) (int, error)

func (*BigRecordsReadStorage) HprofGcObjArrayDumpGetOffset 

func (r *BigRecordsReadStorage) HprofGcObjArrayDumpGetOffset(arrayObjectId core.Identifier) (int, error)

func (*BigRecordsReadStorage) HprofGcPrimArrayDumpGetOffset 

func (r *BigRecordsReadStorage) HprofGcPrimArrayDumpGetOffset(arrayObjectId core.Identifier) (int, error)

type BigRecordsWriteStorage 

type BigRecordsWriteStorage struct {
	// contains filtered or unexported fields
}

func NewBigRecordsWriteStorage 

func NewBigRecordsWriteStorage(
	instanceDumpPersistent io.WriteCloser,
	objArrayDumpPersistent io.WriteCloser,
	primArrayDumpPersistent io.WriteCloser,
) *BigRecordsWriteStorage

func (*BigRecordsWriteStorage) Close 

func (w *BigRecordsWriteStorage) Close() error

func (*BigRecordsWriteStorage) HprofGcInstanceDumpPutOffset 

func (w *BigRecordsWriteStorage) HprofGcInstanceDumpPutOffset(objectId core.Identifier, offset int) error

func (*BigRecordsWriteStorage) HprofGcObjArrayDumpPutOffset 

func (w *BigRecordsWriteStorage) HprofGcObjArrayDumpPutOffset(arrayObjectId core.Identifier, offset int) error

func (*BigRecordsWriteStorage) HprofGcPrimArrayDumpPutOffset 

func (w *BigRecordsWriteStorage) HprofGcPrimArrayDumpPutOffset(arrayObjectId core.Identifier, offset int) error

type Counters 

type Counters struct {
	InstancesCount         map[core.Identifier]int
	PrimArraysCount        map[core.JavaType]int
	PrimArrayElementsCount map[core.JavaType]int
	ObjArraysCount         map[core.Identifier]int
	ObjArrayElementsCount  map[core.Identifier]int
}

type IndexRecordsReadStorage 

type IndexRecordsReadStorage struct {
	// contains filtered or unexported fields
}

IndexRecordsReadStorage uses binary search to read stored index file

func NewIndexRecordsReadStorage 

func NewIndexRecordsReadStorage(persistentStorage IndexRecordsReaderAtCloser, size int) (*IndexRecordsReadStorage, error)

NewIndexRecordsReadStorage opens index file for reading. It assumes that the index file is sorted, it should be checked before using ByteReader.

func (*IndexRecordsReadStorage) Close 

func (db *IndexRecordsReadStorage) Close() error

Close closes the underlying file.

func (*IndexRecordsReadStorage) Get 

func (r *IndexRecordsReadStorage) Get(key uint64) (uint64, error)

Get is used to lookup the offset of the given key in the file. It uses binary search algorithm.

type IndexRecordsReaderAtCloser 

type IndexRecordsReaderAtCloser interface {
	io.ReaderAt
	io.Closer
}

type IndexRecordsWriteCloser 

type IndexRecordsWriteCloser interface {
	io.WriteCloser
}

type IndexRecordsWriteStorage 

type IndexRecordsWriteStorage struct {
	// contains filtered or unexported fields
}

IndexRecordsWriteStorage is the struct that tracks underlying file and currently processed batch.

func NewIndexRecordsWriteStorage 

func NewIndexRecordsWriteStorage(persistentStorage io.WriteCloser, batchSize BatchSize) *IndexRecordsWriteStorage

NewIndexRecordsWriteStorage creates new index file for writing index there. The second argument should be used to specify batch size.

func (*IndexRecordsWriteStorage) Close 

func (w *IndexRecordsWriteStorage) Close() error

Close should be invoked after parsing is over to trigger writing of the active batch.

func (*IndexRecordsWriteStorage) Put 

func (w *IndexRecordsWriteStorage) Put(key uint64, val uint64) error

Put takes one index record and puts it to the current active batch. If batch size equals to the limit it triggers writing to the disk.

func (*IndexRecordsWriteStorage) Write 

func (w *IndexRecordsWriteStorage) Write(batch Batch) error

Write dumps the given batch to the index file

type MetaReadStorage 

type MetaReadStorage struct {
	MetaStorage
}

func NewMetaReadStorage 

func NewMetaReadStorage() *MetaReadStorage

func (*MetaReadStorage) RestoreFrom 

func (s *MetaReadStorage) RestoreFrom(source io.Reader) error

type MetaStorage 

type MetaStorage struct {
	Counters Counters
}

type MetaWriteStorage 

type MetaWriteStorage struct {
	MetaStorage
}

func NewMetaWriteStorage 

func NewMetaWriteStorage() *MetaWriteStorage

func (*MetaWriteStorage) AddInstance 

func (s *MetaWriteStorage) AddInstance(obj any)

func (*MetaWriteStorage) SerializeTo 

func (s *MetaWriteStorage) SerializeTo(destination io.Writer) error

type RamReadVolume 

type RamReadVolume struct {
	*bytes.Reader
}

func NewRamReadVolume 

func NewRamReadVolume(data []byte) *RamReadVolume

func (*RamReadVolume) Close 

func (r *RamReadVolume) Close() error

type RamWriteVolume 

type RamWriteVolume struct {
	*bytes.Buffer
}

func NewRamWriteVolume 

func NewRamWriteVolume() *RamWriteVolume

func (*RamWriteVolume) Close 

func (w *RamWriteVolume) Close() error

type SmallRecordsReadStorage 

type SmallRecordsReadStorage struct {
	// contains filtered or unexported fields
}

func NewSmallRecordsReadStorage 

func NewSmallRecordsReadStorage() *SmallRecordsReadStorage

func (*SmallRecordsReadStorage) GetHprofFrame 

func (s *SmallRecordsReadStorage) GetHprofFrame(stackFrameId core.Identifier) (core.HprofFrame, error)

func (*SmallRecordsReadStorage) GetHprofGcClassDump 

func (s *SmallRecordsReadStorage) GetHprofGcClassDump(classObjectId core.Identifier) (core.HprofGcClassDump, error)

func (*SmallRecordsReadStorage) GetHprofLoadClassByClassObjectId 

func (s *SmallRecordsReadStorage) GetHprofLoadClassByClassObjectId(classObjectId core.Identifier) (core.HprofLoadClass, error)

func (*SmallRecordsReadStorage) GetHprofLoadClassByClassSerialNumer 

func (s *SmallRecordsReadStorage) GetHprofLoadClassByClassSerialNumer(classSerialNumber uint32) (core.HprofLoadClass, error)

func (*SmallRecordsReadStorage) GetHprofTrace 

func (s *SmallRecordsReadStorage) GetHprofTrace(threadSerialNumber uint32) (core.HprofTrace, error)

func (*SmallRecordsReadStorage) GetHprofUtf8 

func (s *SmallRecordsReadStorage) GetHprofUtf8(nameId core.Identifier) (core.HprofUtf8, error)

func (*SmallRecordsReadStorage) ListHprofGcRootJavaFrame 

func (s *SmallRecordsReadStorage) ListHprofGcRootJavaFrame() []core.HprofGcRootJavaFrame

func (*SmallRecordsReadStorage) ListHprofGcRootJniGlobal 

func (s *SmallRecordsReadStorage) ListHprofGcRootJniGlobal() []core.HprofGcRootJniGlobal

func (*SmallRecordsReadStorage) ListHprofGcRootJniLocal 

func (s *SmallRecordsReadStorage) ListHprofGcRootJniLocal() []core.HprofGcRootJniLocal

func (*SmallRecordsReadStorage) ListHprofGcRootStickyClass 

func (s *SmallRecordsReadStorage) ListHprofGcRootStickyClass() []core.HprofGcRootStickyClass

func (*SmallRecordsReadStorage) ListHprofGcRootThreadObj 

func (s *SmallRecordsReadStorage) ListHprofGcRootThreadObj() []core.HprofGcRootThreadObj

func (*SmallRecordsReadStorage) ListHprofLoadClass 

func (s *SmallRecordsReadStorage) ListHprofLoadClass() []core.HprofLoadClass

func (*SmallRecordsReadStorage) RestoreFrom 

func (s *SmallRecordsReadStorage) RestoreFrom(source io.Reader) error

type SmallRecordsWriteStorage 

type SmallRecordsWriteStorage struct {
	// contains filtered or unexported fields
}

func NewSmallRecordsWriteStorage 

func NewSmallRecordsWriteStorage() *SmallRecordsWriteStorage

func (*SmallRecordsWriteStorage) PutHprofFrame 

func (s *SmallRecordsWriteStorage) PutHprofFrame(record core.HprofFrame)

func (*SmallRecordsWriteStorage) PutHprofGcClassDump 

func (s *SmallRecordsWriteStorage) PutHprofGcClassDump(record core.HprofGcClassDump)

func (*SmallRecordsWriteStorage) PutHprofGcRootJavaFrame 

func (s *SmallRecordsWriteStorage) PutHprofGcRootJavaFrame(record core.HprofGcRootJavaFrame)

func (*SmallRecordsWriteStorage) PutHprofGcRootJniGlobal 

func (s *SmallRecordsWriteStorage) PutHprofGcRootJniGlobal(record core.HprofGcRootJniGlobal)

func (*SmallRecordsWriteStorage) PutHprofGcRootJniLocal 

func (s *SmallRecordsWriteStorage) PutHprofGcRootJniLocal(record core.HprofGcRootJniLocal)

func (*SmallRecordsWriteStorage) PutHprofGcRootStickyClass 

func (s *SmallRecordsWriteStorage) PutHprofGcRootStickyClass(record core.HprofGcRootStickyClass)

func (*SmallRecordsWriteStorage) PutHprofGcRootThreadObj 

func (s *SmallRecordsWriteStorage) PutHprofGcRootThreadObj(record core.HprofGcRootThreadObj)

func (*SmallRecordsWriteStorage) PutHprofLoadClass 

func (s *SmallRecordsWriteStorage) PutHprofLoadClass(record core.HprofLoadClass)

func (*SmallRecordsWriteStorage) PutHprofTrace 

func (s *SmallRecordsWriteStorage) PutHprofTrace(record core.HprofTrace)

func (*SmallRecordsWriteStorage) PutHprofUtf8 

func (s *SmallRecordsWriteStorage) PutHprofUtf8(record core.HprofUtf8)

func (*SmallRecordsWriteStorage) PutIdSize 

func (s *SmallRecordsWriteStorage) PutIdSize(idSize uint32)

func (*SmallRecordsWriteStorage) PutTimestamp 

func (s *SmallRecordsWriteStorage) PutTimestamp(timestamp time.Time)

func (*SmallRecordsWriteStorage) SerializeTo 

func (s *SmallRecordsWriteStorage) SerializeTo(destination io.Writer) error

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.