acct_etl

acct_etl

Extracts, transforms and loads accounting data fed by PowerMTA pipe into Redis.

 ./acct_etl -h
Extracts, transforms and loads accounting data fed by PowerMTA pipe into Redis
Usage of ./acct_etl:
  -infile string
        Input file (omit to read from stdin)
  -logfile string
        File written with message logs

Here is an example PowerMTA config file showing "accounting pipe" setup. The pipe carries message attributes that "feeder" uses to augment the open and click event data.

acct_etl internals

You can test without PowerMTA using the included example file:

./acct_etl -infile example.csv

Starting acct_etl, logging to
2020/01/10 19:02:32 PowerMTA accounting headers: [type rcpt header_x-sp-message-id header_x-sp-subaccount-id]
2020/01/10 19:02:32 Loaded acct_headers -> {"header_x-sp-message-id":2,"header_x-sp-subaccount-id":3,"rcpt":1,"type":0} into Redis
2020/01/10 19:02:32 Loaded msgID_0000123456789abcdef0 -> {"header_x-sp-subaccount-id":"0","rcpt":"[email protected]"} into Redis
2020/01/10 19:02:32 Loaded msgID_0000123456789abcdef1 -> {"header_x-sp-subaccount-id":"1","rcpt":"[email protected]"} into Redis
2020/01/10 19:02:32 Loaded msgID_0000123456789abcdef2 -> {"header_x-sp-subaccount-id":"2","rcpt":"[email protected]"} into Redis

The start.sh file copies the acct_etl executable to a place where PowerMTA runs it, and sets owner. It temporarily stops and restarts PowerMTA.

Logfile default location for this process is /opt/pmta/acct_etl.log.

Redis key/value pairs hold data for each message ID, with a configured time-to-live (matching SparkPost's event retention). You can list these keys with redis-cli keys msgID*.