https://maximafoundation.org/ https://gsconsultants.lk/bandarqq/ https://gsconsultants.lk/dominoqq/ slot kamboja slot pulsa slot maxwin pkv games https://revistacipa.com.br/judi-bola-euro-2024/ pkv games https://ashleyadillon.com/ slot77 judi bola slot telkomsel bandarqq https://acapulco.gob.mx/mailer/ https://acapulco.gob.mx/tys/docs/ https://acapulco.gob.mx/pcivil/css/ https://acapulco.gob.mx/pcivil/uac/ https://acapulco.gob.mx/pcivil/recomendaciones/ pkv games dominoqq bonus new member slot depo 5k pkv games slot thailand slot garansi kekalahan slot dana situs toto slot thailand slot gacor mpo play slot bri slot bca slot dana slot mahjong ways slot77 slot77 slot123 pkv games pkv games pkv games pkv games pkv games bandarqq dominoqq dominoqq dominoqq dominoqq MPO mpo play mpo play bandarqq bandarqq pkv games pkv games pkv games jam hoki main slot jam hoki main slot bandarqq bandarqq pkv games bandarqq bandarqq bandarqq bandarqq dominoqq dominoqq dominoqq pkv games pkv games pkv games pokerqq mpo slot indonesia dominoqq bandarqq pkv games bandarqq pkv games pkv games bandarqq dominoqq pkv games pkv games
https://teradata.co.id/dana/ https://cbt.sman21bekasi.sch.id/robopragma/ https://dukcapilbogorkab.id/robopragma/ https://pule-pule.trenggalekkab.go.id/slot-depo-10k/ https://tribratanews.sidoarjo.jatim.polri.go.id/agen-euro/ https://sraten.tuntang.semarangkab.go.id/wp-content/uploads/2023/joker123/ https://sraten.tuntang.semarangkab.go.id/wp-content/uploads/2023/joker123/ https://sraten.tuntang.semarangkab.go.id/wp-content/uploads/2023/sbobet-sbobet88-judi-bola-mix-parlay/ https://bpsk.kuningankab.go.id/indosat/ slot thailand
Skip to content

Help Center

Categories
< All Topics
Print

ElasticSearch Guide

The Elastic stack (or ELK Stack or ElasticSearch) is a framework that hosts, monitors, and populates data following a horizontal NoSQL model. It is used in BT and VT to manage large data volumes and this page acts as an overall guide.

This guide is not intended to be a course on the technology itself, but rather, document how it is used at COSMOS, processes used, etc. To understand the Elastic Stack, there are plenty of resources such as the video below or the slides I presented in the 08/24/2021 meeting.

Server Setup Instructions

For VT, full setup instructions can be found in the repo README here. You will need access to the repo.

Everyday Monitoring

Monitoring is done through the Kibana dashboard (K component of the ELK stack). Accessible locally at http://vtracker-elastic.host.ualr.edu:5601/app/home, using the elastic user (login info in PW sheet).

Monitoring mostly consists of checking the following regularly. For all pages, if there are no results or the page displays an error, make sure to 1. be in the correct dashboard as indicated by the breadcrumbs on top of the below pages, and 2. that the date filter captures enough data.

Metricbeat Hardware Dashboard

Metricbeat Dashboard

This displays general activity, mostly keep an eye on the Disk used. When this gets too high, go in the VM settings and increase the storage size, then allocate it in the server.

Discover Dashboard

Discover Dashboard

This is the “MySQL Workbench” view of Elasticsearch. This is where rows (called documents) can be viewed and queried.

Cluster Dashboard

Elastic Stack Cluster Dashboard

This page shows the dashboard for the entire cluster. It shows an overview of hardware, as well as key components like nodes (which will multiply with a larger cluster), and the Logstash pipeline – the data ingestion.

Logstash Data Pipeline

Logstash Pipeline Overview

This dashboard show which indices (tables) data is being fed into. It is useful to keep track of what is being kept up to date.

Logstash Monitoring

Logstash Pipeline Monitoring

A more in depth look into the Logstash pipeline.

Setting up a Cluster

See Architecture Best Practices. Below is a summary of key hardware components and recommended architecture. See the very comprehensive Benchmarking and sizing your Elasticsearch cluster for logs and metrics for more information.

In ElasticSearch, a cluster contains nodes (machines, physical or virtual) which contains virtual shards. Only one cluster is needed at this stage of production [1]. While machines can be virtual, it is best for them to be separate to insure redundancy.

  1. What architecture to use?

Hot-Warm (Elasticsearch Hot Warm Architecture | Elastic Blog)

“When using elasticsearch for larger time data analytics use cases, we recommend using time-based indices and a tiered architecture with 3 different types of nodes (Master, Hot-Node and Warm-Node), which we refer to as the “Hot-Warm” architecture.”

  1. How many nodes (machines)?
    1. Three (How many nodes should an ElasticSearch cluster have? )
  2. What are the key hardware requirements?
    1. RAM (How many shards should I have in my Elasticsearch cluster? | Elastic Blog)

A good rule-of-thumb is to ensure you keep the number of shards per node below 20 per GB heap it has configured. A node with a 30GB heap should therefore have a maximum of 600 shards, but the further below this limit you can keep it the better. 

(Creating an Elasticsearch Cluster: Getting Started)

 “As a rule of the thumb, the maximum heap size should be set up to 50% of your RAM, but no more than 32GB (due to Java pointer inefficiency in larger heaps).”

  1. CPU – Fig. Current ELK Stack CPU usage

See also the following points, from How to build an elastic search cluster for production? | Cloud Native Computing Foundation

Disks

Disks are probably the most essential aspect of a cluster and especially so for indexing-heavy clusters such as those that ingest log data.

Disks are by far the slowest subsystem in a server. This means that if you have write-heavy loads such as logs retention, you are doing a lot of writing, and the disks can quickly become saturated, which in turn becomes the bottleneck of the cluster.

I highly recommend, if you can afford, to use SSDs. Their far superior writing and reading speed significantly increase your overall performance. SSD-backed nodes see an increase in bot query and indexing performance.

CPU

Let’s talk about the last aspect of hardware performance. CPUs are not so crucial with elastic Search as deployments tend to be relatively light on CPU requirements.

The recommended option is to use a modern processor with multiple cores. Common production-grade ElasticSearch clusters tend to utilize between two to eight-core machines.

If you need to choose between faster CPUs or more cores, choose more cores. The extra concurrency that multiple cores offer will far outweigh a slightly faster clock speed.

Setting up Nodes and Shards

As seen above, a cluster is divided into nodes and shard. To evaluate the architecture that should be used, use this calculator. See its github page for more details. See official documentation for execution.

Table of Contents

© 2024 Collaboratorium for Social Media and Online Behavioral Studies

pkv games pkv games bandarqq dominoqq pkv games bandarqq dominoqq pkv games bandarqq dominoqq pkv games bandarqq dominoqq pkv games bandarqq dominoqq bandarqq pkv games pkv games bandarqq dominoqq slot depo 5k pkv games pkv games bandarqq dominoqq slot depo 5k pkv games pkv games bandarqq dominoqq slot depo 5k pkv games bandarqq pkv games bandarqq dominoqq pkv games bandarqq dominoqq pkv games bandarqq dominoqq pkv games bandarqq pkv games bandarqq dominoqq pkv games dominoqq pkv games bandarqq dominoqq pkv games bandarqq pkv games bandarqq dominoqq pkv games bandarqq dominoqq bonus new member depo 50 bonus 50 slot telkomsel dominoqq bandarqq slot bca dominoqq slot pulsa slot88 bonus new member depo 25 bonus 25 slot garansi kekalahan slot thailand mpo slot depo 10k slot indosat slot zeus pkv pkv games pkv bandarqq dominoqq pkv games bandarqq dominoqq slot depo 5k mpo slot depo 5k demo slot mahjong kakek merah slot bandarqq pkv dominoqq slot depo 5k slot depo 5k depo 25 bonus 25 depo 50 bonus 50 slot77 bonus new member slot garansi kekalahan pkv games slot dana slot dana slot depo 5k slot garansi kekalahan slot gacor malam ini slot ovo slot77 bonus new member depo 25 bonus 25 bonus new member bandarqq dominoqq slot77 slot77 bandarqq pkv slot gacor bandarqq pkv games bandarqq dominoqq slot gacor slot thailand depo 25 bonus 25 slot77 slot depo 5k slot depo 10k slot depo 25k slot depo 5k slot depo 5k slot depo 10k slot depo 25k slot depo 5k slot depo 5k slot depo 10k slot depo 25k slot depo 5k pkv games bandarqq dominoqq slot depo 5k pkv games dominoqq slot depo 5k slot depo 5k slot gacor thailand pkv games bandarqq dominoqq slot depo 5k slot gacor thailand slot gacor thailand slot depo 5k slot77 pkv games bandarqq dominoqq slot depo 5k slot depo 5k bandarqq pkv games dominoqq bandarqq pkv games dominoqq slot depo 5k dominoqq pkv games slot77 bandarqq dominoqq pkv games depo 25 bonus 25 dominoqq bandarqq pkv games rtp slot pkv games bandarqq dominoqq bocoran admin jarwo pkv games bandarqq dominoqq pkv games bandarqq dominoqq Slot x500 pkv games bandarqq dominoqq Slot77 pkv games bandarqq dominoqq pkv games bandarqq pkv games slot gacor slot online slot88 slot demo slot terbaru slot deposit 5000 rtp slot slot pragmatic slot bonanza slot kakek slot malaysia rtp live slot hoki link daftar slot slot x500 slot77 slot777 mpo slot pkv games bandarqq dominoqq pokerqq qq online slot gacor triofus bocoran admin jarwo bocoran admin riki depo 50 bonus 50 depo 25 bonus 25 bonus new member slot slot telkomsel slot xl slot dana slot bca main slot slot thailand slot maxwin link slot link gacor judi parlay judi bola sbobet slot pulsa slot gampang menang slot demo pragmatic slot deposit pulsa slot 5rb slot 10rb slot 25rb slot 50rb slot bonus new member slot e-wallet situs slot slot indonesia slot server luar akun pro slot server thailand slot bni slot bri slot mandiri slot resmi slot linkaja slot server international pola slot slot mahjong slot kamboja slot filipina slot zeus bo slot nexus slot slot mania slot deposit bank slot new bank slot qris Slot Princes slot petir x500 sbobet88 link judi bola bandar judi bola judi bola liga inggris judi bola liga indonesia judi bola liga luar togel sgp idn slot situs live casino togel macau slot gacor hari ini situs slot gacor link slot gacor deposit pulsa tanpa potongan slot casino situs terpercaya situs slot slot terpercaya slot gacor gampang menang slot gacor 2023 slot deposit pulsa tanpa potongan slot gacor malam ini situs gacor hari ini situs slot terpercaya slot pulsa tanpa potongan situs slot gacor hari ini rtp slot gacor slot luar negeri rtp slot gacor hari ini situs slot terbaik slot paling gacor slot tergacor slot gacor slot online slot88 slot terbaru slot deposit 5000 rtp slot slot pragmatic slot bonanza slot kakek slot malaysia rtp live slot hoki link daftar slot slot x500 slot77 slot777 mpo slot dominoqq pokerqq qq online slot gacor triofus bocoran admin jarwo bocoran admin riki depo 50 bonus 50 depo 25 bonus 25 bonus new member slot slot telkomsel slot xl slot dana slot bca main slot slot thailand slot maxwin link slot link gacor judi parlay judi bola sbobet slot pulsa slot gampang menang slot demo pragmatic slot deposit pulsa slot 5rb slot 10rb slot 25rb slot 50rb slot bonus new member slot e wallet situs slot slot indonesia slot server luar akun pro slot server thailand slot bni slot bri slot mandiri slot resmi slot linkaja slot server international pola slot slot mahjong slot kamboja slot filipina slot zeus bo slot nexus slot slot mania slot deposit bank slot new bank slot qris Slot Princes slot petir x500 sbobet88 link judi bola bandar judi bola judi bola liga inggris judi bola liga indonesia judi bola liga luar togel sgp idn slot situs live casino togel macau slot gacor hari ini situs slot gacor link slot gacor deposit pulsa tanpa potongan slot casino situs terpercaya situs slot slot terpercaya slot gacor gampang menang slot gacor 2023 slot deposit pulsa tanpa potongan slot gacor malam ini situs gacor hari ini situs slot terpercaya slot pulsa tanpa potongan situs slot gacor hari ini rtp slot gacor pkv games slot gacor bandarqq pkv games bandarqq dominoqq slot gacor slot online slot88 slot demo slot terbaru slot deposit 5000 rtp slot slot pragmatic slot bonanza slot kakek slot malaysia rtp live slot hoki link daftar slot slot x500 slot77 slot777 mpo slot pkv games bandarqq dominoqq pokerqq qq online slot gacor triofus bocoran admin jarwo bocoran admin riki depo 50 bonus 50 depo 25 bonus 25 bonus new member slot slot telkomsel slot xl slot dana slot bca main slot slot thailand slot maxwin link slot link gacor judi parlay judi bola situs judi bola judi bola terpercaya situs judi bola resmi judi bola parlay situs judi bola resmi di indonesia situs judi bola online agen judi bola terpercaya situs judi bola terbesar di indonesia pasaran judi bola daftar judi bola cara main judi bola pemula web judi bola judi bola dunia prediksi judi bola malam ini daftar judi bola 88 akun judi bola website judi bola situs judi bola resmi dan terpercaya prediksi judi bola pasaran judi bola liga 1 indonesia bandar judi bola terbesar di dunia over under judi bola judi bola indonesia judi bola sbobet slot judi bola judi bola terbaik aplikasi judi bola situs judi bola terbesar di asia daftar situs judi bola resmi judi bola slot situs judi bola resmi asia bandar judi bola situs judi bola terpercaya agen judi bola judi bola online judi bola piala dunia judi bola resmi parlay judi bola link judi bola situs judi bola terbesar sbobet slot pulsa slot gampang menang slot demo pragmatic slot deposit pulsa slot 5rb slot 10rb slot 25rb slot 50rb slot bonus new member slot e-wallet situs slot slot indonesia slot server luar akun pro slot server thailand slot bni slot bri slot mandiri slot resmi slot linkaja slot server international pola slot slot mahjong slot kamboja slot filipina slot zeus bo slot nexus slot slot mania slot deposit bank slot new bank slot qris Slot Princes slot petir x500 sbobet88 link judi bola bandar judi bola judi bola liga inggris judi bola liga indonesia judi bola liga luar togel sgp idn slot situs live casino togel macau slot gacor hari ini situs slot gacor slot demo gacor judi slot gacor triofus link slot gacor slot gacor maxwin gacor x500 situs gacor deposit pulsa tanpa potongan slot casino situs terpercaya situs slot slot terpercaya slot gacor gampang menang slot gacor 2023 slot deposit pulsa tanpapotongan slot gacor malam ini situs gacor hari ini situs slot terpercaya slot pulsa tanpa potongan situs slot gacor hari ini rtp slot gacor slot luar negeri rtp slot gacor hari ini situs slot terbaik slot paling gacor slot tergacor situs gacor malam ini gate of olympus slotdepo 5k rtp gacor rtp gacor hari ini pola gacor olympus deposit pulsa slot server thailand super gacor bo slot gacor link gacor slot info slot gacor gacor slot pola gacor hari ini daftar slot slot online gacor link slot gacor thailand jam gacor slot slot gacor hari ini pragmatic akun slot gacor situs judi slot gacor mpo gacor bocoran slot gacor daftar slot gacor pola slot gacor hari ini slot gacor minimal deposit 5rb pola slot gacor link gacor hari ini info slot gacor hari ini demo slot gacor bocoran slot gacor hari ini slot game situs slot gacor terpercaya slot depo 10k slot gacor terpercaya link slot gacor hari ini slot gacor terbaru daftar slot online slot online terpercaya agen slot thailand akun gacor thailand server gacor thailand agen slot server thailand link slot situs thailand akun slot luar thailand super thailand slot akun slot pro thailand slot akun thailand gacor th