Cosmos-Starmap

Author: Joseph Kready

Introduction

Prometheus is a metric monitoring tool: https://prometheus.io/

• Node Exporter used to get system metrics: https://github.com/prometheus/node_exporter

Grafana is an open source dashboard tool: https://grafana.com/

Loki is log aggerate tool based on Prometheus: https://grafana.com/oss/loki/

• Promtail sends logs to Loki (like node exporter): https://grafana.com/docs/loki/latest/clients/promtail/

Access

Cosmos-Starmap VM

IP Address: 144.167.34.179 (VM located on cosmos-3)

Username: cosmosadmin

Password: Theualrcosmosadminaccount1

Grafana

http://144.167.34.179/    or   http://cosmos-starmap.host.ualr.edu

Username: admin

Password: Theualrcosmosadminaccount1

Promethus

144.167.34.179:9090

You can check the status of different nodes at http://144.167.34.179:9090/targets 

Paths

Promethus config: /etc/prometheus/prometheus.yml

Loki config: /etc/loki/config-loki.yml

Data Pipeline Reference Table

Hostname	IP address (Instance)	Project	Port	Notes
Cosmos- Joseph	144.167.35.211	Transcript Example	8006
Cosmos-Crawler	144.167.35.49	Sentiment & Toxicity	8001
Cosmos-Crawler	144.167.35.49	Blog Transfer	8002
Cosmos-Crawler	144.167.35.49	YT Daily Crawler	8003
Cosmos-Crawler	144.167.35.49	Twitter Crawler	8004
BT-dev1	144.167.35.125	Narrative Ingestion	8001
BT-dev1	144.167.35.125	Blogger Post Processing	8002
BT-dev1	144.167.35.125	Terms Post-processing	8003
BT-dev1	144.167.35.125	Blogsites Post-processing	8004
BT-dev1	144.167.35.125	Entity Sentiment Post-processing	8005
BT-dev1	144.167.35.125	Language Post-processing	8006
BT-dev1	144.167.35.125	Sentiment Post-processing	8007
BT-dev1	144.167.35.125	Toxicity Post-processing	8008
BT-dev1	144.167.35.125	Clustering Post-processing	8009
Cosmos-ytdl-1	144.167.34.58	Youtube video/audio downloader	8001
LIWC (COSMOS-4)	144.167.35.31	LIWC	8001

Adding jobs to Prometheus

1. Login to cosmos-starmap (ssh cosmosadmin@144.167.34.179)
2. sudo nano  /etc/prometheus/prometheus.yml
3. Follow examples in the file to add a job name and it’s targets
4. Save the file by CTR X, Y, ENTER
5. sudo service prometheus restart
6. sudo service prometheus status
   1. If all goes well, the status should be green. 

Getting Logs to Loki using Promtail

Windows:

1. Download the latest release of promtail from https://github.com/grafana/loki/releases/
   1. Get the version for your OS
2. Extract the download to where you want to run Promtail from (I make a folder called Promtail on the root)
   1. Create a sub directory there called ‘tmp’
3. Create a promtail-local-config.yaml file in the same directory (example https://raw.githubusercontent.com/grafana/loki/master/cmd/promtail/promtail-local-config.yaml)
   1. Set the positions  filename to the tmp directory you created earlier
      1. Like this: C:\COSMOS\Promtail\tmp\positions.yaml
   2. For ‘clients: – url:’ set it to: http://144.167.34.179:3100/loki/api/v1/push
   3. Name your project after ‘job:’
   4. Set ‘__path__’ to the path where your log files are stored
      1. Make sure you have ‘\*log’ at the end
   5. *If you have more services on that machine which you want to get logs from, simply copy the promtail yaml file from ‘targets’ down and re-do above. 
4. Make promtail start when your system starts
   1. For windows I create a  bat file that cd into the directory, then does
      1. cd C:\COSMOS\Promtail
      2. start promtail-windows-amd64 –config.file=promtail-local-config.yaml 
   2. Then use task scheduler to make that bat file run on startup with administrator rights
      1. This guide will get you close https://www.howtogeek.com/138159/how-to-enable-programs-and-custom-scripts-to-run-at-boot/
      2. Makes sure to use ‘Run whether user is logged on or not’ and ‘Run with highest Privileges’ 
      3. For Triggers, ‘Begin the task’ – At startup
      4. Under Settings, make sure ‘Stop the task if it runs longer than’ is unchekced

Linux:

I am using this guide: https://sbcode.net/grafana/install-promtail-service/. Make sure you replace the version they use with the most recent version found at https://github.com/grafana/loki/releases/

1. Follow the instructions above until you have created the config-promtail.yml. 
2. Edit the file as so:
   1. For ‘clients: – url:’ set it to: http://144.167.34.179:3100/loki/api/v1/push
   2. Replace everything under ‘scrape_config’ with the scrape_config found here https://raw.githubusercontent.com/grafana/loki/master/cmd/promtail/promtail-local-config.yaml 
   3. Name your project after ‘job:’
   4. Set ‘__path__’ to the path where your log files are stored
      1. Make sure you have ‘/*log’ at the end
   5. *If you have more services on that machine which you want to get logs from, simply copy the promtail yaml file from ‘targets’ down and re-do above. 
3. Follow the rest of the instructions in the guide to Configure promtail as a service
   1. If you see the error in the promtail status msg=”error creating promtail” error=”open /tmp/positions.yaml: permission denied”, run ‘chown promtail:promtail /tmp/positions.yaml’
   2. Skip the last 2 steps
   3. Make sure to run this command at the end: sudo systemctl enable promtail.service
4. We don’t need to configure the firewall, so stop at that point in the guide. Now you should be able to access your log file by the name you gave it. 

*Note: 

By default, promtail won’t pick up the progress bar (tqdm) because it needs a new line between each item in the log file. The cheap solution is to add  postfix=”\n” to the progress bar. You will 

Example: for x in tqdm(video_ids, desc=”Downloading Trans”, file=sys.stdout, also need to set ‘file=sys.stdout’ in the progress bar for it to write to file

postfix=”\n”)]

Python and Prometheus+Grafana

1. Follow the instructions laid out in the github repo: https://github.com/prometheus/client_python to add prometheus to your code
   1. To import metrics into many different modules in your project, I recommend creating a ‘prometheus.py’ file where you store your global variables then import those into each module (referrece: https://stackoverflow.com/questions/15959534/visibility-of-global-variables-in-imported-modules) 
2. If you want to capture logs, make sure to save those log files to a folder using the .log extension. 
   1. Follow the Getting Logs to Loki using Promtail guide
3. Expose ports prometheus will use
   1. To open ports on Ubuntu, just follow https://stackoverflow.com/questions/30251889/how-to-open-some-ports-on-ubuntu 
   2. On windows Open ‘Windows defender Firewall with Advanced Security’ 
   3. Go to Inbound Rules -> New Rule…
   4. Port
   5. TCP, {your port you exposed during start_http_server()}
   6. Allow the connection
   7. All profiles selected
   8. Name it Prometheus
      1. Description; “Ports exposed for monitoring scripts”
   9. Double click on the new rule that was just created
   10. Go to ‘Scope’
   11. For ‘Local IP address’ select ‘Any IP Address’. For ‘Remote IP Address’ select ‘These IP addresses’ and put it to 144.167.34.179 
       1. It might take a minute for prometheus to pickup the new firewall rules. You can check the status at this page. Just wait a minute or 2. http://144.167.34.179:9090/targets
       2. This should now only expose that port to cosmos-starmap
       3. If you have multiple scripts that need to expose prometheus on this computer, you can add them to this firewall rule under ‘Protocols and Ports’ 
   12. Apply and Okay

Capturing Windows Metrics with Windows Exporter

56147

You can follow the instructions here: https://github.com/prometheus-community/windows_exporter

These instructions pair well with this dashboard: https://grafana.com/grafana/dashboards/6593

1. Download the latest release of the windows node Exporter from here (exe file) https://github.com/prometheus-community/windows_exporter/releases
2. Put the exe in a folder. Create a folder called ‘tmp’ in that same place
   1. I like to use C\COSMOS\node_exporter
3. Open powershell as an administrator
4. Run New-Service -name Windows_exporter -displayName Windows_Exporter -binaryPathName “`”C:\\COSMOS\\node_exporter\\windows_exporter-0.15.0-amd64.exe`” –collectors.enabled=`”cpu,cs,logical_disk,net,os,service,system,textfile,tcp,process`” –collector.textfile.directory=`”C:\\COSMOS\\node_exporter\\tmp`””
   1. Make sure to update the ‘binaryPathName’ to the location of your exe file (don’t forget to add the extension)
   2. Update the ‘textfile.directory’ to the location of the tmp folder you created above
5. Expose ports node exporter uses
   1. Node exporter creates a firewall rule already, but we need to change it. On windows open up the ‘Windows defender Firewall with Advanced Security’ 
   2. Search for the firewall rule ‘windows_exporter’, open it
   3. Go to ‘Scope’
   4. For ‘Local IP address’ select ‘Any IP Address’. For ‘Remote IP Address’ select ‘These IP addresses’ and put it to 144.167.34.179 
      1. It might take a minute for prometheus to pickup the new firewall rules. You can check the status at this page. Just wait a minute or 2. http://144.167.34.179:9090/targets
   5. Go to ‘Program and Services’ Click on the ‘browse’ button where it says ‘This program’
   6. Navigate to the node exporter exe file you downloaded and select it (C:\\COSMOS\\node_exporter\\windows_exporter-0.15.0-amd64.exe)
   7. Apply, ok, close
6. Add the new target to cosmos-starmap by following the ‘Adding jobs to Prometheus’ guide above. Put the new target under the ‘windows_server’ section. 

Capturing Linux Metrics with Node Exporter

1. Steps to follow: https://devopscube.com/monitor-linux-servers-prometheus-node-exporter/ 
   1. You will need to update the version of node exporter used in the guide 
      1. You’ll need the ‘linux-amd64’ most likely
   2. ‘:wq’ to exit and save vi
2. Expose the port, sudo ufw allow 9100
3. Add the new target to cosmos-starmap by following the ‘Adding jobs to Prometheus’ guide above. Put the new target under the ‘linux_servers’ section.

Alerting

Great introduction video: https://www.youtube.com/watch?v=n6yZuRr36uI 

*Alerting only works with line graphs!

I like to create a separate panel for alerts on each dashboard. There I will configure the alerts and the slack channels to post them on. 

Errors

• If you created an alert that is retuning this error: error:”1:35: parse error: missing unit character in duration:” this is because you are using $__interval in your main query. Replace that with a fixed interval, like ‘1d’

Extras

• Use ‘locale format’ as the unit type to give numbers commas https://community.grafana.com/t/format-large-numbers-with-commas/1213/5

Debugging

• Sometimes starmap might be acting strangely. You can check the targets  page on grafana and notice the ‘last scrap’ time for the different targets. http://144.167.34.179:9090/targets. The last scrape time should be every few minutes. If you see 10 mins + something is probably wrong
  • To fix, just log into the starmap VM (account info at top of doc)
  • sudo service prometheus restart
  • sudo service prometheus status
  • That should get it working again.