Octeth Training Center
Octeth Training Center
PowerMTA, FluentD and Octeth Bounce Processing

PowerMTA, FluentD and Octeth Bounce Processing

PowerMTA, FluentD and Octeth Bounce Processing

Installation

Install Fluentd to process PowerMTA Logs and post them to Oempro.

Run ulimit -n and if the result is 1024, increase maximum number of file descriptors to 65536:

If your console shows 1024, it is insufficient. Please add the following lines to your /etc/security/limits.conf file and reboot your machine:

root soft nofile 65536
root hard nofile 65536
* soft nofile 65536
* hard nofile 65536

Reboot the server:

reboot

Install Fluentd Ubuntu package by following instructions here.

# Install packages
apt update
apt install gcc make ruby ruby-dev

# Install Fluentd Gem
gem install fluentd --no-doc

Run the following commands to verify the Fluentd installation:

# Setup the fluentd directory and initial configuration
fluentd --setup /etc/fluent

# Run and test
cd /etc/fluent/
fluentd -c /etc/fluent/fluent.conf -vv &
echo '{"json":"message"}' | fluent-cat debug.test

Use kill to stop the test fluentd process.

Boot Setup

The Fluentd gem does not come with /etc/init.d/ scripts. To ensure Fluentd starts on system boot, we need to create a systemd service file for Fluentd:

  1. Create a new service file for Fluentd:
vi /etc/systemd/system/fluentd.service
  1. Paste the following into the file:
[Unit]
Description=Fluentd data collector
After=network.target

[Service]
User=root
ExecStart=/usr/local/bin/fluentd -c /etc/fluent/oempro.conf --under-supervisor
Restart=always

[Install]
WantedBy=multi-user.target

You can learn the actual path of fluentd by executing which fluentd.

  1. Save and close the file.
  2. Reload the systemd daemon to read the new file:
systemctl daemon-reload
  1. Enable Fluentd to start on boot:
systemctl enable fluentd
  1. You can start Fluentd immediately without rebooting by running:
systemctl start fluentd
  1. To check the status of the Fluentd service, you can run:
systemctl status fluentd

This will ensure that Fluentd starts automatically whenever your system boots up.

Configuration

First, make sure that PowerMTA logs properly. Edit the /etc/pmta/config file and make sure that the following directive exists:

# Octeth Logging
<acct-file /var/log/pmta-oempro-accounting/oempro.csv>
    delete-after 40d
    move-interval 24h
    
    records d, b, t, tq, r, f, rb, rs

    record-fields d  *, !dlvEsmtpAvailable, header_From, header_Message-ID, header_Return-Path
    record-fields b  *, !dlvEsmtpAvailable, header_From, header_Message-ID, header_X-FBLId, header_Return-Path
    record-fields t  *, !dlvEsmtpAvailable, header_From, header_Message-ID, header_Return-Path
    record-fields tq *
    record-fields r  *, !srcType, !srcMta, !rcvSourceIp, !rcvDestinationIp, header_From, header_Return-Path
    record-fields f  *, header_X-virtual-MTA, header_X-FBLId, header_Return-Path
    record-fields rb *, header_From, header_X-virtual-MTA, header_X-FBLId, header_Return-Path
    record-fields rs *, header_Return-Path

    map-header-to-field f header_X-HmXmrOriginalRecipient rcpt # JMRP Recipient
</acct-file>

The /etc/fluentd/oempro.conf content should be set to:

<source>
  @type tail
  path /var/log/pmta-oempro-accounting/oempro-*.csv
  pos_file /var/log/pmta_oempro_accounting.pos
  tag pmta.log.oempro

  <parse>
    @type csv
    keys type,timeLogged,timeQueued,orig,rcpt,orcpt,dsnAction,dsnStatus,dsnDiag,dsnMta,bounceCat,srcType,srcMta,dlvType,dlvSourceIp,dlvDestinationIp,dlvSize,vmta,jobId,envId,queue,vmtaPool,header_From,header_Message-ID,header_X-FBLId,repSourceIp,feedbackType,format,userAgent,reportingMta,reportedDomain,header_Return-Path,header_X-job,header_Subject,header_X-virtual-MTA,dsnReportingMta
  </parse>
</source>

<match pmta.log.*>
  @type copy
  <store>
    @type http
    endpoint https://your-octeth-domain.com/system/bounce_webhook?type=fluentd
    open_timeout 2
    http_method post
    content_type json
    json_array true
    <format>
      @type json
    </format>
    <buffer>
      flush_interval 10s
    </buffer>
  </store>
  <store>
    @type stdout
  </store>
</match>

In order to monitor stdout of Fluentd, run this command:

journalctl -u fluentd -f
💠
image

Homepage Customer Area Community Portal Contact Us

Footer Social Icons

©Copyright Octeth, Inc. All rights reserved.