Getting started with the ELK Stack
Looking for a Getting Started With Elk Stack (“Elastic Stack”) guide that shows how to set up the Elastic Stack and get up and running quickly? We’ve referenced the official Elk Guide to help you get started.
First you’ll install the core products:
Then you learn how to implement a system monitoring solution that uses Metricbeat to collect server metrics and ship the data to Elasticsearch, where you can search and visualize the data by using Kibana. After you get the basic setup working, you add Logstash for additional parsing.
To get started, you can install the Elastic Stack on a single VM or even on your laptop.
Elasticsearch is a real-time, distributed storage, search, and analytics engine. It can be used for many purposes, but one context where it excels is indexing streams of semi-structured data, such as logs or decoded network packets.
You can run Elasticsearch on your own hardware, or use the hosted Elasticsearch Service on Elastic Cloud. The Elasticsearch Service is available on both AWS and GCP. Try out the Elasticsearch Service for free. – (we recommend that you try this out, it’s pretty sweet).
curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.4.1-amd64.deb
sudo dpkg -i elasticsearch-7.4.1-amd64.deb
sudo /etc/init.d/elasticsearch start
curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.4.1-x86_64.rpm
sudo rpm -i elasticsearch-7.4.1-x86_64.rpm
sudo service elasticsearch start
For other operating systems, go to the Elasticsearch download page.
Make sure Elasticsearch is up and running
To test that the Elasticsearch daemon is up and running, try sending an HTTP GET request on port 9200.
Kibana is an open source analytics and visualization platform designed to work with Elasticsearch. You use Kibana to search, view, and interact with data stored in Elasticsearch indices. You can easily perform advanced data analysis and visualize your data in a variety of charts, tables, and maps.
We recommend that you install Kibana on the same server as Elasticsearch, but it is not required. If you install the products on different servers, you’ll need to change the URL (IP:PORT) of the Elasticsearch server in the Kibana configuration file, kibana.yml, before starting Kibana.
To download and install Kibana, open a terminal window and use the commands that work with your system:
curl -L -O https://artifacts.elastic.co/downloads/kibana/kibana-7.4.1-linux-x86_64.tar.gz
tar xzvf kibana-7.4.1-linux-x86_64.tar.gz
Launch the Kibana web interface
To launch the Kibana web interface, point your browser to port 5601. For example, http://127.0.0.1:5601.
The Beats are open source data shippers that you install as agents on your servers to send operational data to Elasticsearch. Beats can send data directly to Elasticsearch or via Logstash, where you can further process and enhance the data.
Each Beat is a separately installable product. In this guide, you learn how to install and run Metricbeat with the system module enabled to collect system metrics.
To download and install Metricbeat, open a terminal window and use the commands that work with your system:
curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-7.4.1-amd64.deb
sudo dpkg -i metricbeat-7.4.1-amd64.deb
curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-7.4.1-x86_64.rpm
sudo rpm -vi metricbeat-7.4.1-x86_64.rpm
Ship system metrics to Elasticsearch
Metricbeat provides pre-built modules that you can use to rapidly implement and deploy a system monitoring solution, complete with sample dashboards and data visualizations, in about 5 minutes.
In this section, you learn how to run the system module to collect metrics from the operating system and services running on your server. The system module collects system-level metrics, such as CPU usage, memory, file system, disk IO, and network IO statistics, as well as top-like statistics for every process running on your system.
Before you begin: Verify that Elasticsearch and Kibana are running and that Elasticsearch is ready to receive data from Metricbeat.
To set up the system module and start collecting system metrics:
- From the Metricbeat install directory, enable the system module:
sudo metricbeat modules enable system
- Set up the initial environment:
sudo metricbeat setup -e
- Start Metricbeat:
sudo service metricbeat start
Metricbeat runs and starts sending system metrics to Elasticsearch.
Visualize system metrics in Kibana
To visualize system metrics, open your browser and navigate to the Metricbeat system overview dashboard: http://localhost:5601/app/kibana#/dashboard/Metricbeat-system-overview-ecs
Click Host Overview to see detailed metrics about the selected host.
That’s it! Now you know how to set up the simplest architecture for the Elastic Stack!
In our next blog post, we’ll add Logstash, a tool that allows you to parse, enrich, transform, and buffer data from a variety of sources.