ÔõÑùÔÚLinuxÇéÐÎÖÐʹÓÃLogstash¾ÙÐÐÈÕÖ¾ÆÊÎö£¿
ÔõÑùÔÚlinuxÇéÐÎÖÐʹÓÃlogstash¾ÙÐÐÈÕÖ¾ÆÊÎö£¿
LogstashÊÇÒ»¸öÇ¿Ê¢µÄ¿ªÔ´¹¤¾ß£¬±»ÆÕ±éÓÃÓÚ´¦Öóͷ£ºÍÆÊÎöÖÖÖÖÀàÐ͵ÄÈÕÖ¾Êý¾Ý¡£Ëü¿ÉÒԺܼòÆӵؽ«ÈÕÖ¾Êý¾Ý´Ó²î±ðµÄȪԴÍøÂç¡¢¹ýÂË¡¢×ª»»ºÍ·¢Ë͵½ÖÖÖÖÄ¿µÄµØ¡£±¾ÎĽ«ÏÈÈÝÔõÑùÔÚLinuxÇéÐÎÖÐʹÓÃLogstash¾ÙÐÐÈÕÖ¾ÆÊÎö£¬²¢ÌṩһЩ³£¼ûµÄ´úÂëʾÀý¡£
1. ×°ÖúÍÉèÖÃLogstash
ÔÚ×îÏÈ֮ǰ£¬ÇëÈ·±£ÒѾÔÚLinuxÇéÐÎÖÐ×°ÖÃÁËJavaÔËÐÐʱÇéÐΡ£È»ºó£¬Æ¾Ö¤ÒÔÏ°취װÖúÍÉèÖÃLogstash¡£
ÏÂÔØLogstashµÄѹËõ°ü²¢½âѹµ½Ä¿µÄÎļþ¼Ð£º
wget https://artifacts.elastic.co/downloads/logstash/logstash-7.10.2.tar.gz tar -xzf logstash-7.10.2.tar.gz
µÇ¼ºó¸´ÖÆ
½øÈë½âѹºóµÄÎļþ¼Ð£º
cd logstash-7.10.2
µÇ¼ºó¸´ÖÆ
½¨ÉèÒ»¸öеÄÉèÖÃÎļþlogstash.conf²¢Ð´ÈëÒÔÏÂÄÚÈÝ£º
input { # ÉèÖÃÊäÈëÔ´£¬ÈçÎļþ¡¢ÍøÂçµÈ file { path => "/path/to/your/logfile.log" start_position => "beginning" } } filter { # ÉèÖùýÂËÆ÷£¬Æ¾Ö¤ÐèÇó¾ÙÐйýÂ˺Íת»» grok { match => { "message" => "%{COMBINEDAPACHELOG}" } } } output { # ÉèÖÃÊä³öÄ¿µÄµØ£¬ÈçElasticsearch¡¢ÎļþµÈ elasticsearch { hosts => ["localhost:9200"] index => "mylogs-%{+YYYY.MM.dd}" } }
µÇ¼ºó¸´ÖÆ
ÖµµÃ×¢ÖصÄÊÇ£¬ÉÏÊöÉèÖÃÎļþÖ»ÊÇÒ»¸ö¼òÆÓµÄʾÀý£¬Äã¿ÉÒÔƾ֤×Ô¼ºµÄÐèÇó¾ÙÐÐÏìÓ¦µÄÐ޸ĺÍÀ©Õ¹¡£
Æô¶¯Logstash£º
bin/logstash -f logstash.conf
µÇ¼ºó¸´ÖÆ
È·±£LogstashÀÖ³ÉÆô¶¯£¬²¢¼ì²éÊÇ·ñ½«ÈÕÖ¾Êý¾Ý·¢Ë͵½ÁËÖ¸¶¨µÄÄ¿µÄµØ¡£
2. Logstash³£ÓÃÉèÖÃʾÀý
ÏÂÃæÊÇһЩ³£ÓõÄLogstashÉèÖÃʾÀý£¬ÓÃÓÚʵÏÖ²î±ðµÄ¹¦Ð§ºÍ´¦Öóͷ£ÐèÇó¡£
a. ʹÓÃÕýÔò±í´ïʽÌáÈ¡Òªº¦ÐÅÏ¢
filter { grok { match => { "message" => "%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes}" } } }
µÇ¼ºó¸´ÖÆ
ÉÏÊöÉèÖÃʹÓÃÁËÕýÔò±í´ïʽÀ´ÌáÈ¡ÈÕÖ¾ÖеÄIPµØµã¡¢ÇëÇóÒªÁì¡¢ÇëÇ󷾶ºÍÊý¾Ý¾ÞϸµÈÒªº¦ÐÅÏ¢¡£
b. Ìí¼ÓÌØÁíÍâ×Ö¶Î
filter { mutate { add_field => { "environment" => "dev" } } }
µÇ¼ºó¸´ÖÆ
ÉÏÊöÉèÖý«Ò»¸öÃûΪenvironmentµÄÌØÊâ×Ö¶ÎÌí¼Óµ½Ã¿ÌõÈÕÖ¾¼Í¼ÖУ¬²¢½«ÆäÖµÉèÖÃΪdev¡£
c. ɾ³ýÖ¸¶¨×Ö¶Î
filter { mutate { remove_field => [ "fieldname1", "fieldname2" ] } }
µÇ¼ºó¸´ÖÆ
ÉÏÊöÉèÖý«ÃûΪfieldname1ºÍfieldname2µÄ×ֶδÓÿÌõÈÕÖ¾¼Í¼ÖÐɾ³ý¡£
d. ת»»Ê±¼äÃûÌÃ
filter { date { match => [ "timestamp", "yyyy-MM-dd HH:mm:ss" ] } }
µÇ¼ºó¸´ÖÆ
ÉÏÊöÉèÖý«ÃûΪtimestampµÄ×Ö¶ÎÖеÄʱ¼ä×Ö·û´®×ª»»ÎªÖ¸¶¨µÄÈÕÆÚÃûÌá£
3. ½áÓï
LogstashÊÇÒ»¸ö¹¦Ð§Ç¿Ê¢µÄ¹¤¾ß£¬Äܹ»×ÊÖúÎÒÃÇÔÚLinuxÇéÐÎÖжÔÈÕÖ¾Êý¾Ý¾ÙÐÐÍøÂç¡¢¹ýÂË¡¢×ª»»ºÍ·¢Ë͵ȲÙ×÷¡£±¾ÎÄÏÈÈÝÁËLogstashµÄ×°ÖúÍÉèÖð취£¬²¢ÌṩÁËһЩ³£¼ûµÄÉèÖÃʾÀý¡£Ï£Íûͨ¹ý±¾ÎĵÄÏÈÈÝ£¬ÄãÄܶÔÔõÑùÔÚLinuxÇéÐÎÖÐʹÓÃLogstash¾ÙÐÐÈÕÖ¾ÆÊÎöÓÐËùÏàʶºÍÕÆÎÕ¡£
ÒÔÉϾÍÊÇÔõÑùÔÚLinuxÇéÐÎÖÐʹÓÃLogstash¾ÙÐÐÈÕÖ¾ÆÊÎö£¿µÄÏêϸÄÚÈÝ£¬¸ü¶àÇë¹Ø×¢±¾ÍøÄÚÆäËüÏà¹ØÎÄÕ£¡