Edit
EFK fluent部署安装
1. 环境介绍
Centos 6.5 64bit
2. 安装
td-agent介绍
fluent 考虑到灵活可扩展性,使用Ruby编写,部分功能考虑性能使用C
语言编写。普通用户安装操作Ruby daemon还是有一定难度的。 考虑到flunt的上手难度, fluent专门发布了稳定发布包,就也是所谓的td-agent. td-agent和fluent的区别如下。 新手建议使用td-agent
Step0: 安装准备
I. 优化 File Descriptors
设置 ulimit 执行 ulimit -n
,返回如下:
# ulimit -n > 2014
如果显示的内容是 1024
,则需要修改 /etc/security/limits.conf
添加如下内容
root soft nofile 65536 root hard nofile 65536 * soft nofile 65536 * hard nofile 65536
并执行如下命令
# ulimit -n65535 # ulimit -SHn 65535
II. 优化 Network Kernel 参数
编辑/etc/sysctl.conf
, 并执行 sysctl -w
使其生产。如果环境遇到过 TCP_WAIT 有问题,刚不需要设置如下配置
net.ipv4.tcp_tw_recycle = 1 net.ipv4.tcp_tw_reuse = 1 net.ipv4.ip_local_port_range = 10240 65535
Step1: 从rpm源安装
现在支持CentOS , RHEL5,6,7。 下载并执行 install-redhat-td-agent2.sh 。shell会做两件事情:
curl -L https://toolbelt.treasuredata.com /sh /install-redhat-td-agent2.sh | sh
Step2: 启动td-agent服务
/etc/init.d/td-agent
提供 start,stop,restart
功能
# /etc/init.d/td-agent start Starting td-agent : [确定] # /etc/init.d/td-agent status td-agent is running #ps aux | grep td-agent td-agent 3318 0.0 0.5 223132 20324 ? Sl 11 :39 0 :00 /opt/td-agent /embedded/bin/ruby /usr/sbin/td-agent --log /var/log /td-agent /td-agent .log --use-v1-config --group td-agent --daemon /var/run/td-agent /td-agent .pid td-agent 3321 2.5 1.2 269684 47288 ? Sl 11 :39 0 :00 /opt/td-agent /embedded/bin/ruby /usr/sbin/td-agent --log /var/log /td-agent /td-agent .log --use-v1-config --group td-agent --daemon /var/run/td-agent /td-agent .pid
这里很奇怪,td-agent启动两个进程,从进程名上看也没有能看出这两个进程之间的关系。但如果kill
进程号小的进程,另外一个进程也会退出。如果kill
进程号大的进程,会瞬间被重新拉起
Step3: 验证测试(HTTP方式)
默认 /etc/td-agent/td-agent.conf
默认是从HTTP 取日志并把日志输出到 /var/log/td-agent/td-agent.log
。我们可以通过 curl命令做简单测试
3. 配置td-agent
如果我们已经准备好收集日志了,下面的部分是需要关注的地方 。
配置文件存放
如果Fluentd是通过 td-agent 安装的,config 配置默认存放 /etc/td-agent/td-agent.conf
通过执行如下命令使配置生效
/etc/init.d/td-agent reload
配置文件解析
source directives determine the input sources.
match directives determine the output destinations.
filter directives determine the event processing pipelines.
system directives set system wide configuration.
label directives group the output and filter for internal routing
@include directives include other files.
(1) “source”: where all the data come from Fluentd’s input sources are enabled by selecting and configuring the desired input plugins using source directives. Fluentd’s standard input plugins include http
and forward
. http turns fluentd into an HTTP
endpoint to accept incoming HTTP
messages whereas forward turns fluentd into a TCP
endpoint to accept TCP
packets. Of course, it can be both at the same time (You can add as many sources as you wish):
# Receive events from 24224/tcp # This is used by log forwarding and the fluent-cat command <source > @type forward port 24224 </source ># http://this.host:9880/myapp.access?json={"event":"data"} <source > @type http port 9880 </source >
Each source directive must include a type
parameter. The type
parameter specifies which input plugin to use.
Interlude: Routing
The source
submits events into the Fluentd’s routing engine. An event consists of three entities: tag
, time
and record
. The tag is a string separated by ‘.
’s (e.g. myapp.access), and is used as the directions for Fluentd’s internal routing engine. The time field is specified by input plugins, and it must be in the Unix time format. The record is a JSON object.
Fluentd accepts all non-period characters as a part of a tag. However, since the tag is sometimes used in a different context by output destinations (e.g., table name, database name, key name, etc.), it is strongly recommended that you stick to the lower-case alphabets, digits and underscore , e.g., ^[a-z0-9_]+$
.
In the example above, the HTTP input plugin submits the following event::
# generated by http://this.host:9880/myapp.access?json={"event":"data"} tag: myapp.access time: (current time) record: {"event" :"data" }
(2) “match”: Tell fluentd what to do! The “match” directive looks for events with matching tags and processes them. The most common use of the match directive is to output events to other systems (for this reason, the plugins that correspond to the match directive are called “output plugins”). Fluentd’s standard output plugins include file
and forward
. Let’s add those to our configuration file.
# Receive events from 24224/tcp # This is used by log forwarding and the fluent-cat command <source> @type forward port 24224 </source># http://this.host:9880/myapp.access?json={"event":"data"} <source> @type http port 9880 </source># Match events tagged with "myapp.access" and # store them to /var/log/fluent/access.%Y-%m-%d # Of course, you can control how you partition your data # with the time_slice_format option. <match myapp.access> @type file path /var/log/fluent/access </match>
Each match directive must include a match pattern and a type
parameter. Only events with a tag
matching the pattern will be sent to the output destination (in the above example, only the events with the tag “myapp.access” is matched). The type
parameter specifies the output plugin to use.
Just like input sources, you can add new output destinations by writing your own plugins. For further information regarding Fluentd’s output destinations, please refer to the Output Plugin Overview article.
Match Pattern: how you control the event flow inside fluentd
The following match patterns can be used for the <match>
tag.
*
matches a single tag part.
**
matches zero or more tag parts.
{X,Y,Z}
matches X, Y, or Z, where X, Y, and Z are match patterns.
For example, the pattern {a,b}
matches a
and b
, but does not match c
This can be used in combination with the *
or **
patterns. Examples include a.{b,c}.*
and a.{b,c.**}
When multiple patterns are listed inside one <match>
tag (delimited by one or more whitespaces), it matches any of the listed patterns. For example:
The patterns <match a b>
match a
and b
.
The patterns <match a.** b.*>
match a
, a.b
, a.b.c
. (from the first pattern) and b.d
(from the second pattern).
Match Order
Fluentd tries to match tags in the order that they appear in the config file. So if you have the following configuration:
# ** matches all tags. Bad :( <match **> @type blackhole_plugin </match> <match myapp.access> @type file path /var/log/fluent/access </match>
then myapp.access
is never matched. Wider match patterns should be defined after tight match patterns.
<match myapp.access> @type file path /var/log/fluent/access </match > # Capture all unmatched tags. Good :) <match **> @type blackhole_plugin </match >
If you want to send events to multiple outputs, consider out_copy plugin.
<match myevent.file_and_mongo> @type copy <store> @type file path /var /log /fluent/myapp time_slice_format %Y%m %d time_slice_wait 10m time_format %Y%m %dT%H %M %S%z compress gzip utc </store> <store> @type mongo host fluentd port 27017 database fluentd collection test </store> </match>
(3) “filter”: Event processing pipeline The “filter” directive has same syntax as “match” but “filter” could be chained for processing pipeline. Using filters, event flow is like below:
Input -> filter 1 -> ... -> filter N -> Output
Let’s add standard record_transformer
filter to “match” example.
# http://this.host:9880/myapp.access?json={"event":"data"} <source> @type http port 9880 </source> <filter myapp.access> @type record_transformer <record> host_param "#{Socket.gethostname}" </record> </filter> <match myapp.access> @type file path /var/log/fluent/access </match>
Received event,{"event":"data"}
, goes to record_transformer
filter first. record_transformer
adds “host_param” field to event and filtered event, {"event":"data","host_param":"webserver1"}
, goes to file
output.
You can also add new filters by writing your own plugins. For further information regarding Fluentd’s filter destinations, please refer to the `Filter Plugin Overview article.
Filter’s match order is same as Output and we should put before .
(4) Set system wide configuration: the “system” directive
Following configurations are set by system directive. You can set same configurations by fluentd options::
Here is an example::
<system> # equal to -qq option log_level error # equal to --without-source option without_source # ... </system>
(5) Group filter and output: the “label” directive
The “label” directive groups filter and output for internal routing. “label” reduces the complexity of tag handling.
Here is a configuration example. “label” is built-in plugin parameter so @
prefix is needed.
<source> @type forward </source> <source> @type tail @label @SYSTEM </source> <filter access.**> @type record_transformer <record> # ... </record> </filter> <match **> @type elasticsearch # ... </match> <label @SYSTEM> <filter var.log.middleware.**> @type grep # ... </filter> <match **> @type s3 # ... </match> </label>
In this configuration, forward
events are routed to record_transformer
filter / elasticsearch
output and in_tail
events are routed to grep filter / s3
output inside @SYSTEM
label.
“label” is useful for event flow separation without tag prefix.
ERROR label
@ERROR
label is a built-in label used for error record emitted by plugin’s emit_error_event
API.
If you set <label @ERROR>
in the configuration, events are routed to this label when emit related error, e.g. buffer is full or invalid record.
(6) Re-use your config: the “@include” directive Directives in separate configuration files can be imported using the @include directive::
# Include config files in the ./config.d directory @include config.d /*.conf
The @include directive supports regular file path, glob pattern, and http URL conventions::
# absolute path @include /path/to/config.conf# if using a relative path, the directive will use # the dirname of this config file to expand the path @include extra.conf# glob match pattern @include config.d/*.conf# http @include http: //example.com/fluent .conf
Note for glob pattern, files are expanded in the alphabetical order. If you have a.conf
and b.conf
, fluentd parses a.conf
first. But you should not write the configuration depends on this order. It is so error prone. Please separate @include
for safety.
# If you have a .conf ,b .conf ,...,z .conf and a .conf / z .conf are important... # This is bad @include *.conf # This is good @include a .conf @include config.d /*.conf @include z .conf
Supported Data Types for Values
Each parameter’s type should be documented. If not, please let the plugin author know.
Common plugin parameter
@type
: Specify plugin type
@id
: Specify plugin id. in_monitor_agent uses this value for plugin_id field
@label
: Specify label symbol. See label section
@log_level
: Specify per plugin log level. See Per Plugin Log section
Format tips This section describes useful features in configuration format.
Multi line support for array and hash values
You can write multi line value for array and hash values.
array_param [ "a" , "b" ]hash _param { "k" :"v" , "k1" :10 }
Fluentd assumes [ or { is a start of array / hash. So if you want to set [ or { started but non-json parameter, please use ’ or “. Example1: mail plugin::
<match **> @type mail subject "[CRITICAL] foo's alert system" </match >
Example2: map plugin::
<match tag> @type map map '[["code." + tag, time, { "code" => record["code" ].to_i}], ["time." + tag, time, { "time" => record["time" ].to_i}]]' multi true </match >
We will remove this restriction with configuration parser improvement.
"foo"
is interpreted as foo
, not "foo"
"
is a quote character of string value. It causes the different behaviour between v0.12 and old format in v0.10.
str_param "foo"
In v0.12, str_param is foo
In v0.10 without --use-v1-config
, str_param is "foo"
Embedded Ruby code
Embedded Ruby code
You can evaluate the Ruby code with #{}
in "
quoted string. This is useful for setting machine information like hostname.
host_param "#{Socket.gethostname}" # host_param is actual hostname like `webserver1`.
config-xxx mixins use “${}”, not “#{}”. These embedded configurations are two different things.
In double quoted string literal, \ is escape character
\
is interpreted as escape character. You need \
for setting "
, \r
, \n
, \t
, \
or several characters in double-quoted string literal.
str_param "foo\nbar" # \n is interpreted as actual LF character
@%28%u6280%u672F%u6587%u6863%u5B66%u4E60%29%5B%u6280%u672F%2C%20linux%2C%20%u65E5%u5FD7%5D%0AEFK%20fluent%u90E8%u7F72%u5B89%u88C5%0A%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%0A%0A%23%201.%20%u73AF%u5883%u4ECB%u7ECD%0ACentos%206.5%2064bit%0A%0A%23%202.%20%u5B89%u88C5%0A%23%23%20td-agent%u4ECB%u7ECD%0Afluent%20%u8003%u8651%u5230%u7075%u6D3B%u53EF%u6269%u5C55%u6027%uFF0C%u4F7F%u7528Ruby%u7F16%u5199%uFF0C%u90E8%u5206%u529F%u80FD%u8003%u8651%u6027%u80FD%u4F7F%u7528%60C%60%u8BED%u8A00%u7F16%u5199%u3002%u666E%u901A%u7528%u6237%u5B89%u88C5%u64CD%u4F5CRuby%20daemon%u8FD8%u662F%u6709%u4E00%u5B9A%u96BE%u5EA6%u7684%u3002%0A%u8003%u8651%u5230flunt%u7684%u4E0A%u624B%u96BE%u5EA6%uFF0C%20fluent%u4E13%u95E8%u53D1%u5E03%u4E86%u7A33%u5B9A%u53D1%u5E03%u5305%uFF0C%u5C31%u4E5F%u662F%u6240%u8C13%u7684td-agent.%20td-agent%u548Cfluent%u7684%u533A%u522B%u5982%u4E0B%u3002%20%u65B0%u624B%u5EFA%u8BAE%u4F7F%u7528td-agent%0A%0A%21%5BAlt%20text%5D%28./1458633326587.png%29%0A%0A%23%23%23%20Step0%3A%20%u5B89%u88C5%u51C6%u5907%0A%23%23%23%23%20I.%20%u4F18%u5316%20File%20Descriptors%0A%u8BBE%u7F6E%20ulimit%20%u6267%u884C%20%60ulimit%20-n%60%2C%u8FD4%u56DE%u5982%u4E0B%3A%0A%60%60%60%0A%23%20ulimit%20-n%0A%3E%20%20%20%202014%0A%60%60%60%0A%0A%20%u5982%u679C%u663E%u793A%u7684%u5185%u5BB9%u662F%20%20%601024%60%20%2C%u5219%u9700%u8981%u4FEE%u6539%20%60/etc/security/limits.conf%60%u6DFB%u52A0%u5982%u4E0B%u5185%u5BB9%0A%0A%60%60%60%0Aroot%20soft%20nofile%2065536%0Aroot%20hard%20nofile%2065536%0A*%20soft%20nofile%2065536%0A*%20hard%20nofile%2065536%0A%60%60%60%0A%0A%u5E76%u6267%u884C%u5982%u4E0B%u547D%u4EE4%0A%0A%60%60%60%0A%23%20ulimit%20%20-n%0A65535%0A%23%20ulimit%20-SHn%2065535%0A%60%60%60%0A%0A%23%23%23%23%20II.%20%u4F18%u5316%20Network%20Kernel%20%u53C2%u6570%0A%u7F16%u8F91%60/etc/sysctl.conf%60%20%2C%20%u5E76%u6267%u884C%20%60sysctl%20-w%60%20%u4F7F%u5176%u751F%u4EA7%u3002%u5982%u679C%u73AF%u5883%u9047%u5230%u8FC7%20**TCP_WAIT**%20%u6709%u95EE%u9898%uFF0C%u521A%u4E0D%u9700%u8981%u8BBE%u7F6E%u5982%u4E0B%u914D%u7F6E%0A%60%60%60%0Anet.ipv4.tcp_tw_recycle%20%3D%201%0Anet.ipv4.tcp_tw_reuse%20%3D%201%0Anet.ipv4.ip_local_port_range%20%3D%2010240%20%20%20%2065535%0A%60%60%60%0A%0A%23%23%23%20Step1%3A%20%u4ECErpm%u6E90%u5B89%u88C5%0A%u73B0%u5728%u652F%u6301CentOS%20%uFF0C%20RHEL5%2C6%2C7%u3002%0A%u4E0B%u8F7D%u5E76%u6267%u884C%20%5Binstall-redhat-td-agent2.sh%5D%28https%3A//toolbelt.treasuredata.com/sh/install-redhat-td-agent2.sh%29%20%u3002shell%u4F1A%u505A%u4E24%u4EF6%u4E8B%u60C5%uFF1A%0A*%20%u5B89%u88C5%20%60/etc/yum.repos.d/td.repo%60%0A*%20%u5B89%u88C5%20%60td-agent%60%20rpm%u5305%0A%0A%0A%60%60%60%0Acurl%20-L%20https%3A//toolbelt.treasuredata.com/sh/install-redhat-td-agent2.sh%20%7C%20sh%0A%60%60%60%0A%0A%23%23%23%20Step2%3A%20%u542F%u52A8td-agent%u670D%u52A1%0A%60/etc/init.d/td-agent%60%20%u63D0%u4F9B%20%20%60start%2Cstop%2Crestart%60%u529F%u80FD%0A%60%60%60%0A%23%20/etc/init.d/td-agent%20start%0AStarting%20td-agent%3A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%u786E%u5B9A%5D%0A%23%20/etc/init.d/td-agent%20status%0Atd-agent%20is%20running%0A%23ps%20aux%20%7C%20grep%20td-agent%0Atd-agent%20%203318%20%200.0%20%200.5%20223132%2020324%20%3F%20%20%20%20%20%20%20%20Sl%20%20%2011%3A39%20%20%200%3A00%20/opt/td-agent/embedded/bin/ruby%20/usr/sbin/td-agent%20–log%20/var/log/td-agent/td-agent.log%20–use-v1-config%20–group%20td-agent%20–daemon%20/var/run/td-agent/td-agent.pid%0Atd-agent%20%203321%20%202.5%20%201.2%20269684%2047288%20%3F%20%20%20%20%20%20%20%20Sl%20%20%2011%3A39%20%20%200%3A00%20/opt/td-agent/embedded/bin/ruby%20/usr/sbin/td-agent%20–log%20/var/log/td-agent/td-agent.log%20–use-v1-config%20–group%20td-agent%20–daemon%20/var/run/td-agent/td-agent.pid%0A%60%60%60%0A%3E%20%u8FD9%u91CC%u5F88%u5947%u602A%uFF0Ctd-agent%u542F%u52A8%u4E24%u4E2A%u8FDB%u7A0B%uFF0C%u4ECE%u8FDB%u7A0B%u540D%u4E0A%u770B%u4E5F%u6CA1%u6709%u80FD%u770B%u51FA%u8FD9%u4E24%u4E2A%u8FDB%u7A0B%u4E4B%u95F4%u7684%u5173%u7CFB%u3002%u4F46%u5982%u679C%60kill%60%u8FDB%u7A0B%u53F7%u5C0F%u7684%u8FDB%u7A0B%uFF0C%u53E6%u5916%u4E00%u4E2A%u8FDB%u7A0B%u4E5F%u4F1A%u9000%u51FA%u3002%u5982%u679C%60kill%60%u8FDB%u7A0B%u53F7%u5927%u7684%u8FDB%u7A0B%uFF0C%u4F1A%u77AC%u95F4%u88AB%u91CD%u65B0%u62C9%u8D77%0A%0A%23%23%23%20Step3%3A%20%u9A8C%u8BC1%u6D4B%u8BD5%28HTTP%u65B9%u5F0F%29%0A%u9ED8%u8BA4%20%60/etc/td-agent/td-agent.conf%60%u9ED8%u8BA4%u662F%u4ECE**HTTP**%u53D6%u65E5%u5FD7%u5E76%u628A%u65E5%u5FD7%u8F93%u51FA%u5230%20%60/var/log/td-agent/td-agent.log%60%u3002%u6211%u4EEC%u53EF%u4EE5%u901A%u8FC7%20curl%u547D%u4EE4%u505A%u7B80%u5355%u6D4B%u8BD5%0A%0A%23%203.%20%u914D%u7F6Etd-agent%0A%u5982%u679C%u6211%u4EEC%u5DF2%u7ECF%u51C6%u5907%u597D%u6536%u96C6%u65E5%u5FD7%u4E86%uFF0C%u4E0B%u9762%u7684%u90E8%u5206%u662F%u9700%u8981%u5173%u6CE8%u7684%u5730%u65B9%20%u3002%0A%0A%23%23%20%u914D%u7F6E%u6587%u4EF6%u5B58%u653E%0A%u5982%u679CFluentd%u662F%u901A%u8FC7%20td-agent%20%u5B89%u88C5%u7684%uFF0Cconfig%20%u914D%u7F6E%u9ED8%u8BA4%u5B58%u653E%20%60/etc/td-agent/td-agent.conf%60%20%u901A%u8FC7%u6267%u884C%u5982%u4E0B%u547D%u4EE4%u4F7F%u914D%u7F6E%u751F%u6548%0A%0A%60%60%60%0A/etc/init.d/td-agent%20reload%0A%60%60%60%0A%0A%23%23%20%u914D%u7F6E%u6587%u4EF6%u89E3%u6790%0A1.%20**source**%20directives%20determine%20the%20input%20sources.%0A2.%20**match**%20directives%20determine%20the%20output%20destinations.%0A3.%20**filter**%20directives%20determine%20the%20event%20processing%20pipelines.%0A4.%20**system**%20directives%20set%20system%20wide%20configuration.%0A5.%20**label**%20directives%20group%20the%20output%20and%20filter%20for%20internal%20routing%0A6.%20**@include**%20directives%20include%20other%20files.%0A%0A**%281%29%20%u201Csource%u201D%3A%20where%20all%20the%20data%20come%20from**%0AFluentd%u2019s%20input%20sources%20are%20enabled%20by%20selecting%20and%20configuring%20the%20desired%20input%20plugins%20using%20source%20directives.%20Fluentd%u2019s%20standard%20input%20plugins%20include%20%60http%60%20and%20%60forward%60.%20http%20turns%20fluentd%20into%20an%20%60HTTP%60%20endpoint%20to%20accept%20incoming%20%60HTTP%60%20messages%20whereas%20forward%20turns%20fluentd%20into%20a%20%60TCP%60%20endpoint%20to%20accept%20%60TCP%60%20packets.%20Of%20course%2C%20it%20can%20be%20both%20at%20the%20same%20time%20%28You%20can%20add%20as%20many%20sources%20as%20you%20wish%29%3A%0A%60%60%60%0A%23%20Receive%20events%20from%2024224/tcp%0A%23%20This%20is%20used%20by%20log%20forwarding%20and%20the%20fluent-cat%20command%0A%3Csource%3E%0A%20%20@type%20forward%0A%20%20port%2024224%0A%3C/source%3E%0A%0A%23%20http%3A//this.host%3A9880/myapp.access%3Fjson%3D%7B%22event%22%3A%22data%22%7D%0A%3Csource%3E%0A%20%20@type%20http%0A%20%20port%209880%0A%3C/source%3E%0A%60%60%60%0A%0AEach%20**source**%20directive%20must%20include%20a%20%60type%60%20parameter.%20The%20%60type%60%20parameter%20specifies%20which%20input%20plugin%20to%20use.%0A%0A**Interlude%3A%20Routing**%0A%0AThe%20%60source%60%20submits%20events%20into%20the%20Fluentd%u2019s%20routing%20engine.%20An%20event%20consists%20of%20three%20entities%3A%20**%60tag%60**%2C%20**%60time%60**%20and%20**%60record%60**.%20The%20tag%20is%20a%20string%20separated%20by%20%27%60.%60%u2019s%20%28e.g.%20myapp.access%29%2C%20and%20is%20used%20as%20the%20directions%20for%20Fluentd%u2019s%20internal%20routing%20engine.%20The%20time%20field%20is%20specified%20by%20input%20plugins%2C%20and%20it%20must%20be%20in%20the%20Unix%20time%20format.%20The%20record%20is%20a%20JSON%20object.%0A%3E%20Fluentd%20accepts%20all%20non-period%20characters%20as%20a%20part%20of%20a%20tag.%20However%2C%20since%20the%20tag%20is%20sometimes%20used%20in%20a%20different%20context%20by%20output%20destinations%20%28e.g.%2C%20table%20name%2C%20database%20name%2C%20key%20name%2C%20etc.%29%2C%20**it%20is%20strongly%20recommended%20that%20you%20stick%20to%20the%20lower-case%20alphabets%2C%20digits%20and%20underscore**%2C%20e.g.%2C%20%60%5E%5Ba-z0-9_%5D+%24%60.%0A%0AIn%20the%20example%20above%2C%20the%20HTTP%20input%20plugin%20submits%20the%20following%20event%3A%3A%0A%60%60%60%0A%23%20generated%20by%20http%3A//this.host%3A9880/myapp.access%3Fjson%3D%7B%22event%22%3A%22data%22%7D%0Atag%3A%20myapp.access%0Atime%3A%20%28current%20time%29%0Arecord%3A%20%7B%22event%22%3A%22data%22%7D%0A%60%60%60%0A%0A**%282%29%20%u201Cmatch%u201D%3A%20Tell%20fluentd%20what%20to%20do%21**%0AThe%20%u201Cmatch%u201D%20directive%20looks%20for%20events%20with%20**matching**%20tags%20and%20processes%20them.%20The%20most%20common%20use%20of%20the%20match%20directive%20is%20to%20output%20events%20to%20other%20systems%20%28for%20this%20reason%2C%20the%20plugins%20that%20correspond%20to%20the%20match%20directive%20are%20called%20%u201Coutput%20plugins%u201D%29.%20Fluentd%u2019s%20standard%20output%20plugins%20include%20%60file%60%20and%20%60forward%60.%20Let%u2019s%20add%20those%20to%20our%20configuration%20file.%0A%60%60%60%0A%23%20Receive%20events%20from%2024224/tcp%0A%23%20This%20is%20used%20by%20log%20forwarding%20and%20the%20fluent-cat%20command%0A%3Csource%3E%0A%20%20@type%20forward%0A%20%20port%2024224%0A%3C/source%3E%0A%0A%23%20http%3A//this.host%3A9880/myapp.access%3Fjson%3D%7B%22event%22%3A%22data%22%7D%0A%3Csource%3E%0A%20%20@type%20http%0A%20%20port%209880%0A%3C/source%3E%0A%0A%23%20Match%20events%20tagged%20with%20%22myapp.access%22%20and%0A%23%20store%20them%20to%20/var/log/fluent/access.%25Y-%25m-%25d%0A%23%20Of%20course%2C%20you%20can%20control%20how%20you%20partition%20your%20data%0A%23%20with%20the%20time_slice_format%20option.%0A%3Cmatch%20myapp.access%3E%0A%20%20@type%20file%0A%20%20path%20/var/log/fluent/access%0A%3C/match%3E%0A%60%60%60%0AEach%20**match**%20directive%20must%20include%20a%20match%20pattern%20and%20a%20%60type%60%20parameter.%20Only%20events%20with%20a%20**%60tag%60**%20matching%20the%20pattern%20will%20be%20sent%20to%20the%20output%20destination%20%28in%20the%20above%20example%2C%20only%20the%20events%20with%20the%20tag%20%u201Cmyapp.access%u201D%20is%20matched%29.%20The%20%60type%60%20parameter%20specifies%20the%20output%20plugin%20to%20use.%0A%0AJust%20like%20input%20sources%2C%20you%20can%20add%20new%20output%20destinations%20by%20writing%20your%20own%20plugins.%20For%20further%20information%20regarding%20Fluentd%u2019s%20output%20destinations%2C%20please%20refer%20to%20the%20%5BOutput%20Plugin%20Overview%5D%28http%3A//docs.fluentd.org/articles/output-plugin-overview%29%20article.%0A%0A%0AMatch%20Pattern%3A%20how%20you%20control%20the%20event%20flow%20inside%20fluentd%0A%0A—————%0A%0AThe%20following%20match%20patterns%20can%20be%20used%20for%20the%20%60%3Cmatch%3E%60%20tag.%0A%0A*%20%60*%60%20matches%20a%20single%20tag%20part.%0A%20%20%20%20%20%20%20-%20For%20example%2C%20the%20pattern%20%60a.*%60%20matches%20%60a.b%60%2C%20but%20does%20not%20match%20%60a%60%20or%20%60a.b.c%60%0A*%20%60**%60%20matches%20zero%20or%20more%20tag%20parts.%0A%20%20%20%20%20-%20For%20example%2C%20the%20pattern%20%60a.**%60%20matches%20%60a%60%2C%20%60a.b%60%20and%20%60a.b.c%60%0A*%20%60%7BX%2CY%2CZ%7D%60%20matches%20X%2C%20Y%2C%20or%20Z%2C%20where%20X%2C%20Y%2C%20and%20Z%20are%20match%20patterns.%0A%20%20%20%20-%20For%20example%2C%20the%20pattern%20%60%7Ba%2Cb%7D%60%20matches%20%60a%60%20and%20%60b%60%2C%20but%20does%20not%20match%20%60c%60%0A%20%20%20%20-%20This%20can%20be%20used%20in%20combination%20with%20the%20%60*%60%20or%20%60**%60%20patterns.%20Examples%20include%20%60a.%7Bb%2Cc%7D.*%60%20and%20%60a.%7Bb%2Cc.**%7D%60%0A*%20When%20multiple%20patterns%20are%20listed%20inside%20one%20%60%3Cmatch%3E%60%20tag%20%28delimited%20by%20one%20or%20more%20whitespaces%29%2C%20it%20matches%20any%20of%20the%20listed%20patterns.%20For%20example%3A%0A%20%20%20%20-%20The%20patterns%20%60%3Cmatch%20a%20b%3E%60%20match%20%60a%60%20and%20%60b%60.%0A%20%20%20%20-%20The%20patterns%20%60%3Cmatch%20a.**%20b.*%3E%60%20match%20%60a%60%2C%20%60a.b%60%2C%20%60a.b.c%60.%20%28from%20the%20first%20pattern%29%20and%20%60b.d%60%20%28from%20the%20second%20pattern%29.%0A%0AMatch%20Order%0A%0A——————–%0AFluentd%20tries%20to%20match%20tags%20in%20the%20order%20that%20they%20appear%20in%20the%20config%20file.%20So%20if%20you%20have%20the%20following%20configuration%3A%0A%60%60%60%0A%23%20**%20matches%20all%20tags.%20Bad%20%3A%28%0A%3Cmatch%20**%3E%0A%20%20@type%20blackhole_plugin%0A%3C/match%3E%0A%0A%3Cmatch%20myapp.access%3E%0A%20%20@type%20file%0A%20%20path%20/var/log/fluent/access%0A%3C/match%3E%0A%60%60%60%0Athen%20%60myapp.access%60%20is%20never%20matched.%20Wider%20match%20patterns%20should%20be%20defined%20after%20tight%20match%20patterns.%0A%0A%60%60%60%0A%3Cmatch%20myapp.access%3E%0A%20%20@type%20file%0A%20%20path%20/var/log/fluent/access%0A%3C/match%3E%0A%0A%23%20Capture%20all%20unmatched%20tags.%20Good%20%3A%29%0A%3Cmatch%20**%3E%0A%20%20@type%20blackhole_plugin%0A%3C/match%3E%0A%60%60%60%0A%0AIf%20you%20want%20to%20send%20events%20to%20multiple%20outputs%2C%20consider%20%5Bout_copy%5D%28http%3A//docs.fluentd.org/articles/out_copy%29%20plugin.%0A%0A%60%60%60%0A%3Cmatch%20myevent.file_and_mongo%3E%0A%20%20@type%20copy%0A%20%20%3Cstore%3E%0A%20%20%20%20@type%20file%0A%20%20%20%20path%20/var/log/fluent/myapp%0A%20%20%20%20time_slice_format%20%25Y%25m%25d%0A%20%20%20%20time_slice_wait%2010m%0A%20%20%20%20time_format%20%25Y%25m%25dT%25H%25M%25S%25z%0A%20%20%20%20compress%20gzip%0A%20%20%20%20utc%0A%20%20%3C/store%3E%0A%20%20%3Cstore%3E%0A%20%20%20%20@type%20mongo%0A%20%20%20%20host%20fluentd%0A%20%20%20%20port%2027017%0A%20%20%20%20database%20fluentd%0A%20%20%20%20collection%20test%0A%20%20%3C/store%3E%0A%3C/match%3E%0A%60%60%60%0A%0A**%283%29%20%u201Cfilter%u201D%3A%20Event%20processing%20pipeline**%0AThe%20%u201Cfilter%u201D%20directive%20has%20same%20syntax%20as%20%u201Cmatch%u201D%20but%20%u201Cfilter%u201D%20could%20be%20chained%20for%20processing%20pipeline.%20Using%20filters%2C%20event%20flow%20is%20like%20below%3A%0A%60%60%60%0AInput%20-%3E%20filter%201%20-%3E%20…%20-%3E%20filter%20N%20-%3E%20Output%0A%60%60%60%0ALet%u2019s%20add%20%60standard%20record_transformer%60%20filter%20to%20%u201Cmatch%u201D%20example.%0A%60%60%60%0A%23%20http%3A//this.host%3A9880/myapp.access%3Fjson%3D%7B%22event%22%3A%22data%22%7D%0A%3Csource%3E%0A%20%20@type%20http%0A%20%20port%209880%0A%3C/source%3E%0A%0A%3Cfilter%20myapp.access%3E%0A%20%20@type%20record_transformer%0A%20%20%3Crecord%3E%0A%20%20%20%20host_param%20%22%23%7BSocket.gethostname%7D%22%0A%20%20%3C/record%3E%0A%3C/filter%3E%0A%0A%3Cmatch%20myapp.access%3E%0A%20%20@type%20file%0A%20%20path%20/var/log/fluent/access%0A%3C/match%3E%0A%60%60%60%0AReceived%20event%2C%60%20%7B%22event%22%3A%22data%22%7D%60%2C%20goes%20to%20%60record_transformer%60%20filter%20first.%20%60record_transformer%60%20adds%20%u201Chost_param%u201D%20field%20to%20event%20and%20filtered%20event%2C%20%60%7B%22event%22%3A%22data%22%2C%22host_param%22%3A%22webserver1%22%7D%60%2C%20goes%20to%20%60file%60%20output.%0A%0AYou%20can%20also%20add%20new%20filters%20by%20writing%20your%20own%20plugins.%20For%20further%20information%20regarding%20Fluentd%u2019s%20filter%20destinations%2C%20please%20refer%20to%20the%20%60%5BFilter%20Plugin%20Overview%5D%28http%3A//docs.fluentd.org/articles/filter-plugin-overview%29%20article.%0A%0A%3E%20Filter%27s%20match%20order%20is%20same%20as%20Output%20and%20we%20should%20put%20%3Cfilter%3E%20before%20%3Cmatch%3E.%0A%0A**%284%29%20Set%20system%20wide%20configuration%3A%20the%20%u201Csystem%u201D%20directive**%0A%0AFollowing%20configurations%20are%20set%20by%20system%20directive.%20You%20can%20set%20same%20configurations%20by%20fluentd%20options%3A%3A%0A%0A*%20log_level%0A*%20suppress_repeated_stacktrace%0A*%20emit_error_log_interval%0A*%20suppress_config_dump%0A*%20without_source%0A%0AHere%20is%20an%20example%3A%3A%0A%60%60%60%0A%20%3Csystem%3E%0A%20%20%23%20equal%20to%20-qq%20option%0A%20%20log_level%20error%0A%20%20%23%20equal%20to%20–without-source%20option%0A%20%20without_source%0A%20%20%23%20…%0A%3C/system%3E%0A%60%60%60%0A%0A**%285%29%20Group%20filter%20and%20output%3A%20the%20%u201Clabel%u201D%20directive**%0A%0AThe%20%u201Clabel%u201D%20directive%20groups%20filter%20and%20output%20for%20internal%20routing.%20%u201Clabel%u201D%20reduces%20the%20complexity%20of%20tag%20handling.%0A%0AHere%20is%20a%20configuration%20example.%20%u201Clabel%u201D%20is%20built-in%20plugin%20parameter%20so%20%60@%60%20prefix%20is%20needed.%0A%60%60%60%0A%3Csource%3E%0A%20%20@type%20forward%0A%3C/source%3E%0A%0A%3Csource%3E%0A%20%20@type%20tail%0A%20%20@label%20@SYSTEM%0A%3C/source%3E%0A%0A%3Cfilter%20access.**%3E%0A%20%20@type%20record_transformer%0A%20%20%3Crecord%3E%0A%20%20%20%20%23%20…%0A%20%20%3C/record%3E%0A%3C/filter%3E%0A%3Cmatch%20**%3E%0A%20%20@type%20elasticsearch%0A%20%20%23%20…%0A%3C/match%3E%0A%0A%3Clabel%20@SYSTEM%3E%0A%20%20%3Cfilter%20var.log.middleware.**%3E%0A%20%20%20%20@type%20grep%0A%20%20%20%20%23%20…%0A%20%20%3C/filter%3E%0A%20%20%3Cmatch%20**%3E%0A%20%20%20%20@type%20s3%0A%20%20%20%20%23%20…%0A%20%20%3C/match%3E%0A%3C/label%3E%0A%60%60%60%0A%0AIn%20this%20configuration%2C%20%60forward%60%20events%20are%20routed%20to%20%60record_transformer%60%20filter%20/%20%60elasticsearch%60%20output%20and%20%60in_tail%60%20events%20are%20routed%20to%20grep%20filter%20/%20%60s3%60%20output%20inside%20%60@SYSTEM%60%20label.%0A%0A%u201Clabel%u201D%20is%20useful%20for%20event%20flow%20separation%20without%20tag%20prefix.%0A%0A@ERROR%20label%0A%0A——%0A%60@ERROR%60%20label%20is%20a%20built-in%20label%20used%20for%20error%20record%20emitted%20by%20plugin%u2019s%20%60emit_error_event%60%20API.%0A%0AIf%20you%20set%20%60%3Clabel%20@ERROR%3E%60%20in%20the%20configuration%2C%20events%20are%20routed%20to%20this%20label%20when%20emit%20related%20error%2C%20e.g.%20buffer%20is%20full%20or%20invalid%20record.%0A%0A**%286%29%20Re-use%20your%20config%3A%20the%20%u201C@include%u201D%20directive**%0ADirectives%20in%20separate%20configuration%20files%20can%20be%20imported%20using%20the%20**@include**%20directive%3A%3A%0A%60%60%60%0A%23%20Include%20config%20files%20in%20the%20./config.d%20directory%0A@include%20config.d/*.conf%0A%60%60%60%0AThe%20**@include**%20directive%20supports%20regular%20file%20path%2C%20glob%20pattern%2C%20and%20http%20URL%20conventions%3A%3A%0A%60%60%60%0A%23%20absolute%20path%0A@include%20/path/to/config.conf%0A%0A%23%20if%20using%20a%20relative%20path%2C%20the%20directive%20will%20use%0A%23%20the%20dirname%20of%20this%20config%20file%20to%20expand%20the%20path%0A@include%20extra.conf%0A%0A%23%20glob%20match%20pattern%0A@include%20config.d/*.conf%0A%0A%23%20http%0A@include%20http%3A//example.com/fluent.conf%0A%60%60%60%0ANote%20for%20glob%20pattern%2C%20files%20are%20expanded%20in%20the%20alphabetical%20order.%20If%20you%20have%20%60a.conf%60%20and%20%60b.conf%60%2C%20fluentd%20parses%20%60a.conf%60%20first.%20But%20you%20should%20not%20write%20the%20configuration%20depends%20on%20this%20order.%20It%20is%20so%20error%20prone.%20Please%20separate%20%60@include%60%20for%20safety.%0A%60%60%60%0A%23%20If%20you%20have%20a.conf%2Cb.conf%2C…%2Cz.conf%20and%20a.conf%20/%20z.conf%20are%20important…%0A%0A%23%20This%20is%20bad%0A@include%20*.conf%0A%0A%23%20This%20is%20good%0A@include%20a.conf%0A@include%20config.d/*.conf%0A@include%20z.conf%0A%60%60%60%0A**Supported%20Data%20Types%20for%20Values**%0A%3EEach%20parameter%27s%20type%20should%20be%20documented.%20If%20not%2C%20please%20let%20the%20plugin%20author%20know.%0A%0A**Common%20plugin%20parameter**%0A*%20%60@type%60%3A%20Specify%20plugin%20type%0A*%20%60@id%60%3A%20Specify%20plugin%20id.%20in_monitor_agent%20uses%20this%20value%20for%20plugin_id%20field%0A*%20%60@label%60%3A%20Specify%20label%20symbol.%20See%20label%20section%0A*%20%60@log_level%60%3A%20Specify%20per%20plugin%20log%20level.%20See%20Per%20Plugin%20Log%20section%0A%0A**Format%20tips**%0AThis%20section%20describes%20useful%20features%20in%20configuration%20format.%0A%0AMulti%20line%20support%20for%20array%20and%20hash%20values%0A%0A——%0AYou%20can%20write%20multi%20line%20value%20for%20array%20and%20hash%20values.%0A%60%60%60%0Aarray_param%20%5B%0A%20%20%22a%22%2C%20%22b%22%0A%5D%0Ahash_param%20%7B%0A%20%20%22k%22%3A%22v%22%2C%0A%20%20%22k1%22%3A10%0A%7D%0A%60%60%60%0A%0AFluentd%20assumes%20%5B%20or%20%7B%20is%20a%20start%20of%20array%20/%20hash.%20So%20if%20you%20want%20to%20set%20%5B%20or%20%7B%20started%20but%20non-json%20parameter%2C%20please%20use%20%27%20or%20%22.%0AExample1%3A%20mail%20plugin%3A%3A%0A%60%60%60%0A%3Cmatch%20**%3E%0A%20%20@type%20mail%0A%20%20subject%20%22%5BCRITICAL%5D%20foo%27s%20alert%20system%22%0A%3C/match%3E%0A%60%60%60%0AExample2%3A%20map%20plugin%3A%3A%0A%60%60%60%0A%3Cmatch%20tag%3E%0A%20%20@type%20map%0A%20%20map%20%27%5B%5B%22code.%22%20+%20tag%2C%20time%2C%20%7B%20%22code%22%20%3D%3E%20record%5B%22code%22%5D.to_i%7D%5D%2C%20%5B%22time.%22%20+%20tag%2C%20time%2C%20%7B%20%22time%22%20%3D%3E%20record%5B%22time%22%5D.to_i%7D%5D%5D%27%0A%20%20multi%20true%0A%3C/match%3E%0A%60%60%60%0A%3E%09We%20will%20remove%20this%20restriction%20with%20configuration%20parser%20improvement.%0A%0A%60%22foo%22%60%20is%20interpreted%20as%20%60foo%60%2C%20not%20%60%22foo%22%60%0A%0A—–%0A%60%22%60%20is%20a%20quote%20character%20of%20string%20value.%20It%20causes%20the%20different%20behaviour%20between%20v0.12%20and%20old%20format%20in%20v0.10.%0A%0A%20%20%20%20str_param%20%22foo%22%0A%0A*%20In%20v0.12%2C%20str_param%20is%20%60foo%60%0A*%20In%20v0.10%20without%20%60–use-v1-config%60%2C%20str_param%20is%20%60%22foo%22%60%0A%0AEmbedded%20Ruby%20code%0A%0A—-%0A%0A%0AEmbedded%20Ruby%20code%0A%0A——————-%0AYou%20can%20evaluate%20the%20Ruby%20code%20with%20%60%23%7B%7D%60%20in%20%60%22%60%20quoted%20string.%20This%20is%20useful%20for%20setting%20machine%20information%20like%20hostname.%0A%0A%20%20%20%20host_param%20%22%23%7BSocket.gethostname%7D%22%20%23%20host_param%20is%20actual%20hostname%20like%20%60webserver1%60.%0A%0A%3Econfig-xxx%20mixins%20use%20%22%24%7B%7D%22%2C%20not%20%22%23%7B%7D%22.%20These%20embedded%20configurations%20are%20two%20different%20things.%0A%0AIn%20double%20quoted%20string%20literal%2C%20%5C%20is%20escape%20character%0A%0A—————-%0A%0A%60%5C%60%20is%20interpreted%20as%20escape%20character.%20You%20need%20%60%5C%60%20for%20setting%20%60%22%60%2C%20%60%5Cr%60%2C%20%60%5Cn%60%2C%20%60%5Ct%60%2C%20%60%5C%60%20or%20several%20characters%20in%20double-quoted%20string%20literal.%0A%0A%20%20%20%20str_param%20%22foo%5Cnbar%22%20%23%20%5Cn%20is%20interpreted%20as%20actual%20LF%20character%0A%0A%0A%23%23%20Application%20Log%u6536%u96C6%28Ruby%2Cjava%2CPython%2CPHp%2CPerl%2CNode.js%2CScala%29%0A%0A%23%23%20%u6848%u4F8B%20%u6536%u96C6Apache%20%u65E5%u5FD7%u5230MongoDb
原创文章,作者:stanley,如若转载,请注明出处:http://www.178linux.com/13316