README
¶
Watermill Benchmark
This is an early set of tools for benchmarking watermill.
Warning: This tool is meant to provide a rough estimate on how fast each Pub/Sub can process messages. It uses very simplified infrastructure to set things up and default configurations.
Keep in mind that final performance depends on multiple factors.
How it works
- All tests are run on a single 16 CPU GCloud compute instance (
n1-highcpu-16). - Docker Compose is used to run Pub/Sub infrastructure and benchmark code (except for Google Cloud Pub/Sub).
- The tool will first produce a big number of messages on a generated topic.
- Then it will subscribe to the topic and consume all of the messages.
- Multiple message sizes can be chosen (by default: 16, 64 and 256 bytes).
Results
This is an early version of benchmark results. Expect it to be updated and grow.
Kafka (one node)
| Message size (bytes) | Publish (messages / s) | Subscribe (messages / s) |
|---|---|---|
| 16 | 44,090 | 108.285 |
| 64 | 41,371 | 108,848 |
| 256 | 41,497 | 111,756 |
NATS Jetstream (16 Subscribers)
| Message size (bytes) | Publish (messages / s) | Subscribe (messages / s) | Subscribe (messages / s - async ack) |
|---|---|---|---|
| 16 | 49,255 | 33,009 | 63,065 |
| 64 | 49,296 | 33,016 | 62,667 |
| 256 | 48.488 | 32,573 | 62,745 |
NATS Jetstream (48 Subscribers)
| Message size (bytes) | Publish (messages / s) | Subscribe (messages / s) | Subscribe (messages / s - async ack) |
|---|---|---|---|
| 16 | 48,882 | 45,275 | 90,917 |
| 64 | 48,681 | 44,746 | 89,527 |
| 256 | 48,097 | 44,487 | 90,510 |
Redis
| Message size (bytes) | Publish (messages / s) | Subscribe (messages / s) |
|---|---|---|
| 16 | 61,642 | 11,213 |
| 64 | 58,554 | 11,252 |
| 256 | 58,906 | 11,408 |
SQL (MySQL)
| Message size (bytes) | Publish (messages / s) | Subscribe (messages / s) |
|---|---|---|
| 16 | 5,599 | 167 |
| 64 | 5,625 | 168 |
| 256 | 5,381 | 164 |
SQL (PostgreSQL)
| Message size (bytes) | Publish (messages / s) | Subscribe (messages / s) |
|---|---|---|
| 16 | 3,834 | 455 |
| 64 | 3,923 | 468 |
| 256 | 3,855 | 472 |
Google Cloud Pub/Sub (16 subscribers)
| Message size (bytes) | Publish (messages / s) | Subscribe (messages / s) |
|---|---|---|
| 16 | 3,689 | 30,229 |
| 64 | 3,408 | 26,448 |
| 256 | 6,967 | 30,123 |
AMQP (RabbitMQ, 16 subscribers)
| Message size (bytes) | Publish (messages / s) | Subscribe (messages / s) |
|---|---|---|
| 16 | 2,702 | 13,192 |
| 64 | 2,712 | 12,980 |
| 256 | 2,692 | 8,027 |
GoChannel
| Message size (bytes) | Publish (messages / s) | Subscribe (messages / s) |
|---|---|---|
| 16 | 331,882 | 118,943 |
| 64 | 298,847 | 123,499 |
| 256 | 373,053 | 130,940 |
VM Setup
The project includes Terraform definition for setting up an instance on Google Cloud Platform.
It will spin up a fresh Ubuntu 19.04 instance, install docker with dependencies and clone this repository.
Set environment variables:
# path to GCP credentials file
TF_VAR_credentials_path=
# project name on GCP
TF_VAR_project=
# public part of the key that you will use to access SSH
TF_VAR_pub_key_path=
Create the VM:
cd setup
terraform apply
The command will output the public IP address of the server. Use ssh with user benchmark to access it.
After running all benchmarks, destroy the VM:
terraform destroy
Configuration
Google Pub/Sub
Set environment variables in compose/.env:
# path to json file within the project with GCP credentials
GOOGLE_APPLICATION_CREDENTIALS=compose/key.json
# project name on GCP
GOOGLE_CLOUD_PROJECT=
Running
Run benchmarks with:
./run.sh <pubsub>
Directories