DarkMetrix Log
A fast network logging system powered by Apache Kafka.
Architecture
Quick start
Preperation
An Apache Kafka need to be installed previously and create a topic.
./kafka-topics.sh --zookeeper localhost:2181 --create --replication-factor 1 --partitions 10 --topic net_log
See DarkMetrix/log/agent or DarkMetrix/log/server to get more information about Kafka setting.
Install from source
git clone [email protected]:DarkMetrix/log.git
Build agent & run (You will need a root user)
$cd DarkMetrix/log/agent/src
$go build -o ../bin/dm_log_agent
$../admin/start.sh
Build server & run
$cd DarkMetrix/log/server/src
$go build -o ../bin/dm_log_server
$../admin/start.sh
Protocol
We use google/protobuf as the protocol, so you shuold use protobuf whatever programming language you use to marshal the log and send via unix domain socket.
log.proto
syntax = "proto2";
package log_proto;
//Log package
message LogPackage
{
optional string project = 1; //Project name(Also used as the folder name)
optional string service = 2; //Service name(Also used in the log file name)
optional uint32 level = 3; //Log level(INFO = 0, WARNING = 1, ERROR = 2, FATAL = 3)
optional bytes log = 4; //Log content
};
Configuration
All configuration file is in json.
agent
See DarkMetrix/log/agent to get more information.
server
See DarkMetrix/log/server to get more information.
Lisense
DarkMetrix Log
MIT license
Dependencies