Fluentd on Dockerの構築

ログ収集を行うためにfluentdを使用するが、その前提となる環境構築メモ。

環境構築

公式を参考に進める。

.
├── docker-compose.yml
└── fluentd
    └── etc
        └── fluent.conf

docker-compose.yml

version: '3'
services:
  fluentd:
    image: fluent/fluentd:v1.11-1
    volumes:
      - ./fluentd/etc:/fluentd/etc

fluent.confは、とりあえずHello worldをやってみるため、こちらを参考に設定する。

<source>
  @type dummy
  dummy {"hello":"world"}
</source>

実行

Dockerを起動させてみる。

docker-compose up

fluentd_1  | 2020-07-24 13:02:19 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
fluentd_1  | 2020-07-24 13:02:19 +0000 [info]: gem 'fluentd' version '1.11.1'
fluentd_1  | 2020-07-24 13:02:19 +0000 [error]: config error in:
fluentd_1  | <source>
fluentd_1  |   @type dummy
fluentd_1  |   dummy {"hello":"world"}
fluentd_1  | </source>
fluentd_1  |
fluentd_1  | 2020-07-24 13:02:19 +0000 [error]: config error file="/fluentd/etc/fluent.conf" error_class=Fluent::ConfigError error="'tag' parameter is required"
fluentd_fluentd_1 exited with code 1

エラーになった…

再実行

エラーメッセージを見るとtagが必須とのこと。

fluent.confを次のように修正した。

<source>
  @type dummy
  dummy {"hello":"world"}
  tag hello_world
</source>

修正後、Dockerを起動する。

docker-compose up

fluentd_1  | 2020-07-24 13:04:10 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
fluentd_1  | 2020-07-24 13:04:10 +0000 [info]: gem 'fluentd' version '1.11.1'
fluentd_1  | 2020-07-24 13:04:10 +0000 [warn]: both of Plugin @id and path for <storage> are not specified. Using on-memory store.
fluentd_1  | 2020-07-24 13:04:10 +0000 [warn]: both of Plugin @id and path for <storage> are not specified. Using on-memory store.
fluentd_1  | 2020-07-24 13:04:10 +0000 [info]: using configuration file: <ROOT>
fluentd_1  |   <source>
fluentd_1  |     @type dummy
fluentd_1  |     dummy {"hello":"world"}
fluentd_1  |     tag "hello_world"
fluentd_1  |   </source>
fluentd_1  | </ROOT>
fluentd_1  | 2020-07-24 13:04:10 +0000 [info]: starting fluentd-1.11.1 pid=6 ruby="2.5.8"
fluentd_1  | 2020-07-24 13:04:10 +0000 [info]: spawn command to main:  cmdline=["/usr/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/bin/fluentd", "-c", "/fluentd/etc/fluent.conf", "-p", "/fluentd/plugins", "--under-supervisor"]
fluentd_1  | 2020-07-24 13:04:10 +0000 [info]: adding source type="dummy"
fluentd_1  | 2020-07-24 13:04:10 +0000 [warn]: #0 both of Plugin @id and path for <storage> are not specified. Using on-memory store.
fluentd_1  | 2020-07-24 13:04:10 +0000 [warn]: #0 both of Plugin @id and path for <storage> are not specified. Using on-memory store.
fluentd_1  | 2020-07-24 13:04:10 +0000 [info]: #0 starting fluentd worker pid=20 ppid=6 worker=0
fluentd_1  | 2020-07-24 13:04:10 +0000 [info]: #0 fluentd worker is now running worker=0
fluentd_1  | 2020-07-24 13:04:11 +0000 [warn]: #0 no patterns matched tag="hello_world"
fluentd_1  | 2020-07-24 13:04:12 +0000 [warn]: #0 no patterns matched tag="hello_world"
fluentd_1  | 2020-07-24 13:04:14 +0000 [warn]: #0 no patterns matched tag="hello_world"
fluentd_1  | 2020-07-24 13:04:18 +0000 [warn]: #0 no patterns matched tag="hello_world"

エラーはなくなったが[warn]: #0 no patterns matched tag="hello_world"と表示される。

これは、インプットに対してのアウトプットが存在しないことが原因の模様。

そこで、fluent.confを次のように修正する。

<source>
  @type dummy
  dummy {"hello":"world"}
  tag hello_world
</source>

<match hello_world.**>
  @type stdout
</match>

修正後、Dockerを起動する。

docker-compose up

fluentd_1  | 2020-07-24 13:07:38 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
fluentd_1  | 2020-07-24 13:07:39 +0000 [info]: gem 'fluentd' version '1.11.1'
fluentd_1  | 2020-07-24 13:07:39 +0000 [warn]: both of Plugin @id and path for <storage> are not specified. Using on-memory store.
fluentd_1  | 2020-07-24 13:07:39 +0000 [warn]: both of Plugin @id and path for <storage> are not specified. Using on-memory store.
fluentd_1  | 2020-07-24 13:07:39 +0000 [info]: using configuration file: <ROOT>
fluentd_1  |   <source>
fluentd_1  |     @type dummy
fluentd_1  |     dummy {"hello":"world"}
fluentd_1  |     tag "hello_world"
fluentd_1  |   </source>
fluentd_1  |   <match hello_world.**>
fluentd_1  |     type stdout
fluentd_1  |   </match>
fluentd_1  | </ROOT>
fluentd_1  | 2020-07-24 13:07:39 +0000 [info]: starting fluentd-1.11.1 pid=6 ruby="2.5.8"
fluentd_1  | 2020-07-24 13:07:39 +0000 [info]: spawn command to main:  cmdline=["/usr/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/bin/fluentd", "-c", "/fluentd/etc/fluent.conf", "-p", "/fluentd/plugins", "--under-supervisor"]
fluentd_1  | 2020-07-24 13:07:39 +0000 [warn]: #0 'type' is deprecated parameter name. use '@type' instead.
fluentd_1  | 2020-07-24 13:07:39 +0000 [info]: adding match pattern="hello_world.**" type="stdout"
fluentd_1  | 2020-07-24 13:07:39 +0000 [info]: adding source type="dummy"
fluentd_1  | 2020-07-24 13:07:39 +0000 [warn]: #0 both of Plugin @id and path for <storage> are not specified. Using on-memory store.
fluentd_1  | 2020-07-24 13:07:39 +0000 [warn]: #0 both of Plugin @id and path for <storage> are not specified. Using on-memory store.
fluentd_1  | 2020-07-24 13:07:39 +0000 [info]: #0 starting fluentd worker pid=20 ppid=6 worker=0
fluentd_1  | 2020-07-24 13:07:39 +0000 [info]: #0 fluentd worker is now running worker=0
fluentd_1  | 2020-07-24 13:07:40.084009100 +0000 hello_world: {"hello":"world"}
fluentd_1  | 2020-07-24 13:07:41.001777200 +0000 hello_world: {"hello":"world"}
fluentd_1  | 2020-07-24 13:07:42.021463500 +0000 hello_world: {"hello":"world"}
fluentd_1  | 2020-07-24 13:07:43.037985700 +0000 hello_world: {"hello":"world"}

期待した内容が出力されるようになった。

最後に

fluentdがなんとなく動くことが確認できた。

最終的にはログをElasticsearchやS3に置いていきたいので引き続き調査を進める。

零からの

ゆるーく綴るブログ

Fluentd on Dockerの構築

環境構築

実行

再実行

最後に