Introduction
Logstash is a log management tool that collects data from a variety of sources, transforms it on the fly, and sends it to your desired destination. It is often used as a data pipeline for Elasticsearch, an open-source analytics and search engine.
Filebeat is a lightweight shipper for forwarding and centralizing log data. Once installed as an agent, it monitors the log files or locations you specify, collects log events, and forwards them for indexing, usually to Elasticsearch or Logstash.
Now, Manticore also supports the use of Logstash and Filebeat as processing pipelines. This allows the collected and transformed data to be sent to Manticore just like to Elasticsearch. Currently, Logstash versions >= 7.6 and Filebeat versions >= 7.10 are supported.
Let’s examine a simple example of a Logstash config file used for indexing dpkg.log, a standard log file of the Debian package manager. The log itself has a simple structure, as shown below:
2023-05-31 10:42:55 status triggers-awaited ca-certificates-java:all 20190405ubuntu1.1
2023-05-31 10:42:55 trigproc libc-bin:amd64 2.31-0ubuntu9.9 <none>
2023-05-31 10:42:55 status half-configured libc-bin:amd64 2.31-0ubuntu9.9
2023-05-31 10:42:55 status installed libc-bin:amd64 2.31-0ubuntu9.9
2023-05-31 10:42:55 trigproc systemd:amd64 245.4-4ubuntu3.21 <none>
Logstash configuration
Here is an example Logstash configuration:
input {
file {
path => ["/var/log/dpkg.log"]
start_position => "beginning"
sincedb_path => "/dev/null"
mode => "read"
exit_after_read => "true"
file_completed_action => "log"
file_completed_log_path => "/dev/null"
}
}
output {
elasticsearch {
index => " dpkg_log"
hosts => ["http://localhost:9308"]
ilm_enabled => false
manage_template => false
}
}
Note that, before proceeding further, one crucial caveat needs to be addressed: Manticore does not support Log Template Management and the Index Lifecycle Management features of Elasticsearch. As these features are enabled by default in Logstash, they need to be explicitly disabled in the config. Additionally, the hosts option in the output config section must correspond to Manticore’s HTTP listen port (default is localhost:9308).
Logstash results
After adjusting the config as described, you can run Logstash, and the data from the dpkg log will be passed to Manticore and properly indexed.
Here is the resulting schema of the created table and an example of the inserted document:
mysql> DESCRIBE dpkg_log;
+------------------+--------+---------------------+
| Field | Type | Properties |
+------------------+--------+---------------------+
| id | bigint | |
| message | text | indexed stored |
| @version | text | indexed stored |
| @timestamp | text | indexed stored |
| path | text | indexed stored |
| host | text | indexed stored |
+------------------+--------+---------------------+
mysql> SELECT * FROM dpkg_log LIMIT 1\G
*************************** 1. row ***************************
id: 7280000849080746110
host: logstash-db848f65f-lnlf9
message: 2023-04-12 02:03:21 status unpacked libc-bin:amd64 2.31-0ubuntu9
path: /var/log/dpkg.log
@timestamp: 2023-06-16T09:23:57.405Z
@version: 1
Filebeat configuration
An alternative way of collecting raw data is by using a Filebeat agent. Below is a Filebeat config to work with our example dpkg log:
filebeat.inputs:
- type: filestream
id: example
paths:
- /var/log/dpkg.log
output.elasticsearch:
hosts: ["http://localhost:9308"]
index: "dpkg_log"
allow_older_versions: true
setup.ilm:
enabled: false
setup.template:
name: "dpkg_log"
pattern: "dpkg_log"
Configuration for Filebeat versions higher than 8.10
Filebeat versions higher than 8.10 have the output compression feature enabled by default. That is why the compression_level: 0 option must be added to the configuration file to provide compatibility with Manticore:
filebeat.inputs:
- type: filestream
id: example
paths:
- /var/log/dpkg.log
output.elasticsearch:
hosts: ["http://localhost:9308"]
index: "dpkg_log"
allow_older_versions: true
compression_level: 0
setup.ilm:
enabled: false
setup.template:
name: "dpkg_log"
pattern: "dpkg_log"
Filebeat results
Once you run Filebeat with this configuration, log data will be sent to Manticore and properly indexed. Here is the resulting schema of the table created by Manticore and an example of the inserted document:
mysql> DESCRIBE dpkg_log;
+------------------+--------+--------------------+
| Field | Type | Properties |
+------------------+--------+--------------------+
| id | bigint | |
| @timestamp | text | indexed stored |
| message | text | indexed stored |
| log | json | |
| input | json | |
| ecs | json | |
| host | json | |
| agent | json | |
+------------------+--------+--------------------+
mysql> SELECT * FROM dpkg_log LIMIT 1\G
*************************** 1. row ***************************
id: 7280000849080753116
@timestamp: 2023-06-16T09:27:38.792Z
message: 2023-04-12 02:06:08 status half-installed libhogweed5:amd64 3.5.1+really3.5.1-2
input: {"type":"filestream"}
ecs: {"version":"1.6.0"}
host: {"name":"logstash-db848f65f-lnlf9"}
agent: {"ephemeral_id":"587c2ebc-e7e2-4e27-b772-19c611115996","id":"2e3d985b-3610-4b8b-aa3b-2e45804edd2c","name":"logstash-db848f65f-lnlf9","type":"filebeat","version":"7.10.0","hostname":"logstash-db848f65f-lnlf9"}
log: {"offset":80,"file":{"path":"/var/log/dpkg.log"}}
Conclusion
Thus, you can now easily index your log data using Manticore in conjunction with Logstash or Filebeat. The integration of Manticore with Logstash and Filebeat opens up new opportunities for easily indexing log data.