Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,5 +4,5 @@

[Fluentd](https://www.fluentd.org/) is an open-source data collector for a unified logging layer. Fluentd allows you to unify data collection and consumption for better use and understanding of data.

[Fluentd](https://www.fluentd.org/) is licensed under the terms of the [Apache License v2.0](http://www.apache.org/licenses/LICENSE-2.0). This project is made and sponsored by [Treasure Data](https://www.treasuredata.com).
[Fluentd](https://www.fluentd.org/) is licensed under the terms of the [Apache License v2.0](https://www.apache.org/licenses/LICENSE-2.0). This project is made and sponsored by [Treasure Data](https://www.treasuredata.com).

2 changes: 1 addition & 1 deletion container-deployment/docker-logging-driver.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Docker Logging Driver

The article describes how to implement a unified logging system for your [Docker](http://www.docker.com) containers. An application in a production environment needs to register certain events or problems during its runtime.
The article describes how to implement a unified logging system for your [Docker](https://www.docker.com) containers. An application in a production environment needs to register certain events or problems during its runtime.

The old-fashioned way is to write these messages into a log file, but that inherits certain problems. Specifically, when we try to perform some analysis over the registers, or on the other hand, if the application has multiple instances running, the scenario becomes even more complex.

Expand Down
2 changes: 1 addition & 1 deletion container-deployment/install-by-docker.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Docker Image

This article explains how to use the official [Fluentd Docker image](https://hub.docker.com/r/fluent/fluentd/), maintained by [Treasure Data, Inc](http://www.treasuredata.com/).
This article explains how to use the official [Fluentd Docker image](https://hub.docker.com/r/fluent/fluentd/), maintained by [Treasure Data, Inc](https://www.treasuredata.com/).

* [Fluentd official Docker image](https://hub.docker.com/r/fluent/fluentd/)
* [Fluentd official Docker image \(Source\)](https://github.com/fluent/fluentd-docker-image)
Expand Down
2 changes: 1 addition & 1 deletion container-deployment/kubernetes.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

![Fluentd + Kubernetes](../.gitbook/assets/fluentd_kubernetes.png)

[Kubernetes](http://kubernetes.io) provides two logging endpoints for applications and cluster logs:
[Kubernetes](https://kubernetes.io) provides two logging endpoints for applications and cluster logs:

1. Stackdriver Logging for use with Google Cloud Platform; and,
2. Elasticsearch.
Expand Down
2 changes: 1 addition & 1 deletion deployment/plugin-management.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ You should specify the target version with `-v` option:

## `--gemfile` option

A Ruby application manages gem dependencies using Gemfile and [`Bundler`](http://bundler.io/). Fluentd's `--gemfile` option takes the same approach, and is useful for managing plugin versions separated from shared gems.
A Ruby application manages gem dependencies using Gemfile and [`Bundler`](https://bundler.io/). Fluentd's `--gemfile` option takes the same approach, and is useful for managing plugin versions separated from shared gems.

For example, if you have the following Gemfile at `/etc/fluent/Gemfile`:

Expand Down
2 changes: 1 addition & 1 deletion deployment/trouble-shooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ If `fluentd` suddenly hits an unexpected high CPU usage problem, there are sever
* regular expression with broken data
* system calls have a bug, e.g. `inotify` with lots of files

In such cases, you can use `perf` tool on recent Linux to investigate the problem. See [Linux perf Examples](http://www.brendangregg.com/perf.html) page. If you want to know which call causes the problem, [`pid2line.rb`](https://gist.github.com/nurse/0619b6af90df140508c2) is useful.
In such cases, you can use `perf` tool on recent Linux to investigate the problem. See [Linux perf Examples](https://www.brendangregg.com/perf.html) page. If you want to know which call causes the problem, [`pid2line.rb`](https://gist.github.com/nurse/0619b6af90df140508c2) is useful.

## Check Uncaught Logs

Expand Down
2 changes: 1 addition & 1 deletion filter/geoip.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,7 +140,7 @@ Set backend library.

#### Plot Realtime Access Statistics on a World Map using Elasticsearch and Kibana

The `country_code` field is needed to visualize access statistics on a world map using [Kibana](http://www.elasticsearch.org/overview/kibana/).
The `country_code` field is needed to visualize access statistics on a world map using [Kibana](https://www.elasticsearch.org/overview/kibana/).

Required plugins:

Expand Down
8 changes: 4 additions & 4 deletions how-to-guides/apache-to-mongodb.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,9 @@ This article explains how to use [Fluentd](https://fluentd.org/)'s MongoDB Outpu

## Background

[Fluentd](https://fluentd.org/) is an advanced open-source log collector originally developed at [Treasure Data, Inc](http://www.treasuredata.com/). Because Fluentd handles logs as semi-structured data streams, the ideal database should have strong support for semi-structured data. Several candidates meet this criterion, but we believe [MongoDB](http://www.mongodb.org/) is the market leader.
[Fluentd](https://fluentd.org/) is an advanced open-source log collector originally developed at [Treasure Data, Inc](https://www.treasuredata.com/). Because Fluentd handles logs as semi-structured data streams, the ideal database should have strong support for semi-structured data. Several candidates meet this criterion, but we believe [MongoDB](https://www.mongodb.org/) is the market leader.

MongoDB is an open-source, document-oriented database developed at [MongoDB, Inc](http://www.mongodb.com/). It is schema-free and uses a JSON-like format to manage semi-structured data.
MongoDB is an open-source, document-oriented database developed at [MongoDB, Inc](https://www.mongodb.com/). It is schema-free and uses a JSON-like format to manage semi-structured data.

This article will show you how to use [Fluentd](https://fluentd.org/) to import Apache logs into MongoDB.

Expand All @@ -30,7 +30,7 @@ Fluentd does these three \(3\) things:
The following software/services are required to be set up correctly:

* [Fluentd](https://www.fluentd.org/)
* [MongoDB](http://www.mongodb.org/)
* [MongoDB](https://www.mongodb.org/)
* [Apache](https://httpd.apache.org/) (with the Combined Log Format)

For simplicity, this article will describe how to set up a one-node configuration.
Expand All @@ -42,7 +42,7 @@ You can install Fluentd via major packaging systems.

For MongoDB, please refer to the following downloads page:

* [MongoDB Downloads](http://www.mongodb.org/downloads)
* [MongoDB Downloads](https://www.mongodb.org/downloads)

## Install MongoDB Plugin

Expand Down
2 changes: 1 addition & 1 deletion how-to-guides/apache-to-s3.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ This article explains how to use [Fluentd](https://fluentd.org/)'s Amazon S3 Out

## Background

[Fluentd](https://fluentd.org/) is an advanced open-source log collector originally developed at [Treasure Data, Inc](http://www.treasuredata.com/). One of the main objectives of log aggregation is data archiving. [Amazon S3](http://aws.amazon.com/s3/), the cloud object storage provided by Amazon, is a popular solution for data archiving.
[Fluentd](https://fluentd.org/) is an advanced open-source log collector originally developed at [Treasure Data, Inc](https://www.treasuredata.com/). One of the main objectives of log aggregation is data archiving. [Amazon S3](https://aws.amazon.com/s3/), the cloud object storage provided by Amazon, is a popular solution for data archiving.

This article will show you how to use [Fluentd](https://fluentd.org/) to import Apache logs into Amazon S3.

Expand Down
8 changes: 4 additions & 4 deletions how-to-guides/http-to-hdfs.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
# Data Collection with Hadoop \(HDFS\)

This article explains how to use [Fluentd](https://fluentd.org/)'s [WebHDFS Output plugin](http://github.com/fluent/fluent-plugin-webhdfs/) to aggregate semi-structured logs into Hadoop HDFS.
This article explains how to use [Fluentd](https://fluentd.org/)'s [WebHDFS Output plugin](https://github.com/fluent/fluent-plugin-webhdfs/) to aggregate semi-structured logs into Hadoop HDFS.

## Background

[Fluentd](https://fluentd.org/) is an advanced open-source log collector originally developed at [Treasure Data, Inc](http://www.treasuredata.com/). Fluentd is specifically designed to solve the big-data log collection problem. A lot of users are using Fluentd with MongoDB, and have found that it doesn't scale well for now.
[Fluentd](https://fluentd.org/) is an advanced open-source log collector originally developed at [Treasure Data, Inc](https://www.treasuredata.com/). Fluentd is specifically designed to solve the big-data log collection problem. A lot of users are using Fluentd with MongoDB, and have found that it doesn't scale well for now.

HDFS \(Hadoop\) is a natural alternative for storing and processing a huge amount of data. It supports an HTTP interface called WebHDFS in addition to its Java library.

Expand Down Expand Up @@ -128,14 +128,14 @@ drwxr-xr-x - 1 supergroup 0 2012-10-22 09:40 /log/20121022_14/access.

## Conclusion

Fluentd with WebHDFS makes the realtime log collection simple, robust and scalable! [@tagomoris](http://github.com/tagomoris) has already been using this plugin to collect 20,000 msgs/sec, 1.5 TB/day without any major problems for several months now.
Fluentd with WebHDFS makes the realtime log collection simple, robust and scalable! [@tagomoris](https://github.com/tagomoris) has already been using this plugin to collect 20,000 msgs/sec, 1.5 TB/day without any major problems for several months now.

## Learn More

* [Fluentd Architecture](https://www.fluentd.org/architecture)
* [Fluentd Get Started](../quickstart/)
* [WebHDFS Output Plugin](../output/webhdfs.md)
* [Slides: Fluentd and WebHDFS](http://www.slideshare.net/tagomoris/fluentd-and-webhdfs)
* [Slides: Fluentd and WebHDFS](https://www.slideshare.net/tagomoris/fluentd-and-webhdfs)

If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](https://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License.

4 changes: 2 additions & 2 deletions how-to-guides/http-to-td.md
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,7 @@ Result :
+-----+
```

It is not advisable to send sensitive user information to the cloud. To assist with this need, `out_tdlog` comes with some anonymization systems. For more details, see [Treasure Data plugin](http://github.com/treasure-data/fluent-plugin-td/).
It is not advisable to send sensitive user information to the cloud. To assist with this need, `out_tdlog` comes with some anonymization systems. For more details, see [Treasure Data plugin](https://github.com/treasure-data/fluent-plugin-td/).

## Conclusion

Expand All @@ -139,7 +139,7 @@ Fluentd + Treasure Data gives you a data collection and analysis system in days,
* [Fluentd Architecture](https://www.fluentd.org/architecture)
* [Fluentd Get Started](../quickstart/)
* [Treasure Data](https://www.fluentd.org/treasuredata)
* [Treasure Data: Documentation](http://docs.treasuredata.com/)
* [Treasure Data: Documentation](https://docs.treasuredata.com/)

If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](https://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License.

4 changes: 2 additions & 2 deletions how-to-guides/kinesis-stream.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ This article explains how to use [Fluentd](https://fluentd.org/)'s [Amazon Kines

## Background

[Fluentd](https://fluentd.org/) is an advanced open-source log collector originally developed at [Treasure Data, Inc](http://www.treasuredata.com/). Because Fluentd can collect logs from various sources, [Amazon Kinesis](https://aws.amazon.com/kinesis/) is one of the popular destinations for the output.
[Fluentd](https://fluentd.org/) is an advanced open-source log collector originally developed at [Treasure Data, Inc](https://www.treasuredata.com/). Because Fluentd can collect logs from various sources, [Amazon Kinesis](https://aws.amazon.com/kinesis/) is one of the popular destinations for the output.

Amazon Kinesis is a platform for streaming data on AWS, offering powerful services to make it easy to load and analyze streaming data, and also providing the ability for you to build custom streaming data applications for specialized needs.

Expand Down Expand Up @@ -122,7 +122,7 @@ The `random_partition_key true` option will generate the partition key via UUID

For additional configurations, see [Kinesis Output plugin](https://github.com/awslabs/aws-fluent-plugin-kinesis).

For those who are interested in security, all communication between Fluentd and Amazon Kinesis are done via HTTPS. If you do not want to have AES keys in the configuration file, [IAM Role-based authentication](http://docs.aws.amazon.com/kinesis/latest/dev/controlling-access.html) is available too for EC2 nodes.
For those who are interested in security, all communication between Fluentd and Amazon Kinesis are done via HTTPS. If you do not want to have AES keys in the configuration file, [IAM Role-based authentication](https://docs.aws.amazon.com/kinesis/latest/dev/controlling-access.html) is available too for EC2 nodes.

## Test

Expand Down
4 changes: 2 additions & 2 deletions how-to-guides/raspberrypi-cloud-data-logger.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Cloud Data Logging with Raspberry Pi

[Raspberry Pi](http://www.raspberrypi.org/) is a credit-card-sized single-board computer. Because it is low-cost and easy to equip with various types of sensors, using Raspberry Pi as a cloud data logger is one of its ideal use cases.
[Raspberry Pi](https://www.raspberrypi.org/) is a credit-card-sized single-board computer. Because it is low-cost and easy to equip with various types of sensors, using Raspberry Pi as a cloud data logger is one of its ideal use cases.

![Cloud Data Logger by Raspberry Pi](../.gitbook/assets/raspberry-pi-cloud-data-logger.png)

Expand All @@ -10,7 +10,7 @@ This article introduces how to transport sensor data from Raspberry Pi to the cl

[Raspbian](http://www.raspbian.org/) is a free operating system based on Debian, optimized for the Raspberry Pi. Please install Raspbian on your Raspberry Pi by following the instructions in the blog post below:

* [Getting Started with Raspberry Pi: Installing Raspbian](http://www.andrewmunsell.com/blog/getting-started-raspberry-pi-install-raspbian)
* [Getting Started with Raspberry Pi: Installing Raspbian](https://www.andrewmunsell.com/blog/getting-started-raspberry-pi-install-raspbian)

## Install Fluentd

Expand Down
2 changes: 1 addition & 1 deletion input/exec.md
Original file line number Diff line number Diff line change
Expand Up @@ -184,7 +184,7 @@ If you already have a script that runs periodically \(say, via `cron`\) that you

The only requirement for the script is that it outputs TSV, JSON or MessagePack.

For example, this [script](https://gist.github.com/kiyoto/1bd903ad1bdd6ac51fcc) scrapes the front page of [Hacker News](http://news.ycombinator.com) and scrapes information about each post:
For example, this [script](https://gist.github.com/kiyoto/1bd903ad1bdd6ac51fcc) scrapes the front page of [Hacker News](https://news.ycombinator.com) and scrapes information about each post:

Suppose that script is called `hn.rb`. Then, you can run it every 5 minutes with the following configuration:

Expand Down
2 changes: 1 addition & 1 deletion input/forward.md
Original file line number Diff line number Diff line change
Expand Up @@ -337,7 +337,7 @@ The array of usernames.

## Protocol

This plugin accepts both JSON and [MessagePack](http://msgpack.org/) messages and automatically detects which one is used. Internally, Fluentd uses MessagePack as it is more efficient than JSON.
This plugin accepts both JSON and [MessagePack](https://msgpack.org/) messages and automatically detects which one is used. Internally, Fluentd uses MessagePack as it is more efficient than JSON.

The time value is an `EventTime` or a platform-specific integer and is based on the output of Ruby's `Time.now.to_i` function. On Linux, BSD, and Mac systems, this is the number of seconds since 1970.

Expand Down
4 changes: 2 additions & 2 deletions input/syslog.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ This tells Fluentd to create a socket listening on port 5140. You need to set up

### Example Usage

The retrieved data is organized as follows. Fluentd's tag is generated by the `tag` parameter \(tag prefix\), [facility level](http://en.wikipedia.org/wiki/Syslog#Facility_Levels), and [priority](http://en.wikipedia.org/wiki/Syslog#Severity_levels). The record is parsed by the `regexp` [here](https://github.com/fluent/fluentd/blob/master/lib/fluent/plugin/parser_syslog.rb#L23).
The retrieved data is organized as follows. Fluentd's tag is generated by the `tag` parameter \(tag prefix\), [facility level](https://en.wikipedia.org/wiki/Syslog#Facility_Levels), and [priority](https://en.wikipedia.org/wiki/Syslog#Severity_levels). The record is parsed by the `regexp` [here](https://github.com/fluent/fluentd/blob/master/lib/fluent/plugin/parser_syslog.rb#L23).

```text
tag = "#{@tag}.#{facility}.#{priority}"
Expand Down Expand Up @@ -57,7 +57,7 @@ The value must be `syslog`.

### `tag` \(required\)

The prefix of the tag. The tag itself is generated by the tag prefix, [facility level](http://en.wikipedia.org/wiki/Syslog#Facility_Levels), and [priority](http://en.wikipedia.org/wiki/Syslog#Severity_levels).
The prefix of the tag. The tag itself is generated by the tag prefix, [facility level](https://en.wikipedia.org/wiki/Syslog#Facility_Levels), and [priority](https://en.wikipedia.org/wiki/Syslog#Severity_levels).

### `port`

Expand Down
2 changes: 1 addition & 1 deletion installation/before-install.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ net.ipv4.ip_local_reserved_ports = 24224

Use `sysctl -p` command or reboot your node for the changes to take effect.

These kernel options were originally taken from the presentation [How Netflix Tunes EC2 Instances for Performance](https://www.slideshare.net/brendangregg/how-netflix-tunes-ec2-instances-for-performance) by [Brendan Gregg](http://www.brendangregg.com/), Senior Performance Architect at AWS re:Invent 2017.
These kernel options were originally taken from the presentation [How Netflix Tunes EC2 Instances for Performance](https://www.slideshare.net/brendangregg/how-netflix-tunes-ec2-instances-for-performance) by [Brendan Gregg](https://www.brendangregg.com/), Senior Performance Architect at AWS re:Invent 2017.

## Use sticky bit symlink/hardlink protection

Expand Down
4 changes: 2 additions & 2 deletions installation/install-by-deb-td-agent-v3.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
# Install by DEB Package v3 \(Debian/Ubuntu\)

This article explains how to install the `td-agent` deb package, the stable Fluentd distribution package maintained by [Treasure Data, Inc](http://www.treasuredata.com/).
This article explains how to install the `td-agent` deb package, the stable Fluentd distribution package maintained by [Treasure Data, Inc](https://www.treasuredata.com/).

## What is `td-agent`?

Fluentd is written in Ruby for flexibility, with performance-sensitive parts in C. However, some users may have difficulty installing and operating a Ruby daemon.

That is why [Treasure Data, Inc](http://www.treasuredata.com/) provides **the stable distribution of Fluentd**, called `td-agent`. The differences between Fluentd and `td-agent` can be found [here](https://www.fluentd.org/faqs).
That is why [Treasure Data, Inc](https://www.treasuredata.com/) provides **the stable distribution of Fluentd**, called `td-agent`. The differences between Fluentd and `td-agent` can be found [here](https://www.fluentd.org/faqs).

## Installing `td-agent`

Expand Down
4 changes: 2 additions & 2 deletions installation/install-by-deb-td-agent-v4.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
# Install by DEB Package \(Debian/Ubuntu\)

This article explains how to install stable versions of `td-agent` deb packages, the stable Fluentd distribution packages maintained by [Treasure Data, Inc](http://www.treasuredata.com/).
This article explains how to install stable versions of `td-agent` deb packages, the stable Fluentd distribution packages maintained by [Treasure Data, Inc](https://www.treasuredata.com/).

## What is `td-agent`?

Fluentd is written in Ruby for flexibility, with performance-sensitive parts in C. However, some users may have difficulty installing and operating a Ruby daemon.

That is why [Treasure Data, Inc](http://www.treasuredata.com/) provides **the stable distribution of Fluentd**, called `td-agent`. The differences between Fluentd and `td-agent` can be found [here](https://www.fluentd.org/faqs).
That is why [Treasure Data, Inc](https://www.treasuredata.com/) provides **the stable distribution of Fluentd**, called `td-agent`. The differences between Fluentd and `td-agent` can be found [here](https://www.fluentd.org/faqs).

This installation guide is for `td-agent` v4. `td-agent` v4 uses fluentd v1 in the core. See [this page](../appendix/td-agent-v2-vs-v3-vs-v4.md) for the comparison and supported OS.

Expand Down
Loading