Introduction to Monitoring and Logging

Effective monitoring and logging in Kubernetes are essential for understanding cluster performance, diagnosing issues, and maintaining reliability. Kubernetes provides native tools and integrates with popular solutions to collect metrics, visualize data, and log application and system events.

Why Monitoring and Logging Are Crucial

1. Real-Time Insights: Understand the state of your cluster, Pods, and applications.
2. Issue Diagnosis: Quickly identify and resolve performance bottlenecks or failures.
3. Proactive Alerts: Detect and address problems before they impact end users.
4. Compliance: Maintain logs for audits and compliance requirements.

Key Concepts in Monitoring and Logging

1. Metrics: Quantitative data about resource usage (CPU, memory, etc.).
2. Logs: Time-stamped records of application/system events.
3. Tracing: Tracks the flow of requests through distributed systems.
4. Monitoring Tools: Solutions like Prometheus, Grafana, and Kubernetes Dashboard.
5. Logging Tools: Tools such as Fluentd, Elasticsearch, and Kibana (EFK stack).

Step-by-Step Implementation

Step 1: Setting Up Prometheus and Grafana

Prometheus is an open-source monitoring system that collects metrics from Kubernetes clusters. Grafana visualizes these metrics.

Install Prometheus

1. Add Helm Repository:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

2. Install Prometheus:

helm install prometheus prometheus-community/kube-prometheus-stack --namespace monitoring --create-namespace

3. Verify Installation:

kubectl get pods -n monitoring

Access Prometheus Dashboard

1. Forward the Prometheus server port:

kubectl port-forward svc/prometheus-kube-prometheus-prometheus -n monitoring 9090:9090

2. Open your browser and go to http://localhost:9090.

Install Grafana

1. Check Grafana Pod:

kubectl get pods -n monitoring | grep grafana

2. Access Grafana: Forward the Grafana port:

kubectl port-forward svc/prometheus-grafana -n monitoring 3000:80

3. Open http://localhost:3000 in your browser.

4. Login Credentials:

• Username: admin
• Password: Retrieve it from the Kubernetes secret:bashCopy code

kubectl get secret -n monitoring prometheus-grafana -o jsonpath="{.data.admin-password}" | base64 --decode

5. Add Prometheus as a Data Source:

• In Grafana, go to Configuration > Data Sources > Add Data Source.
• Select Prometheus and configure it using http://prometheus-kube-prometheus-prometheus.monitoring.svc:9090.

Step 2: Monitoring Kubernetes with Metrics Server

The Metrics Server provides resource usage data for Pods and Nodes.

Install Metrics Server

1. Apply Metrics Server YAML:

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

2. Verify Installation:

kubectl get deployment metrics-server -n kube-system

3. View Metrics:

kubectl top nodes
kubectl top pods

Step 3: Centralized Logging with EFK Stack

The EFK stack (Elasticsearch, Fluentd, Kibana) is widely used for centralized logging in Kubernetes.

Deploy Elasticsearch

1. Add the Helm repository:

helm repo add elastic https://helm.elastic.co
helm repo update

2. Install Elasticsearch:

helm install elasticsearch elastic/elasticsearch --namespace logging --create-namespace

Deploy Fluentd

1. Create a ConfigMap for Fluentd:

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-config
  namespace: logging
data:
  fluent.conf: |
    <source>
      @type tail
      path /var/log/containers/*.log
      pos_file /var/log/fluentd-containers.log.pos
      tag kubernetes.*
      format json
    </source>
    <match kubernetes.**>
      @type elasticsearch
      host elasticsearch-master
      port 9200
      logstash_format true
    </match>

Apply the ConfigMap:

kubectl apply -f fluentd-config.yaml

2. Deploy Fluentd:

kubectl apply -f https://raw.githubusercontent.com/fluent/fluentd-kubernetes-daemonset/master/fluentd-daemonset-elasticsearch-rbac.yaml

Deploy Kibana

1. Install Kibana:

helm install kibana elastic/kibana --namespace logging

2. Forward Kibana Port:

kubectl port-forward svc/kibana-kibana -n logging 5601:5601

3. Open Kibana Dashboard: Visit http://localhost:5601 in your browser.

Step 4: Troubleshooting Logs

1. View logs for a specific Pod:

kubectl logs <pod-name>

2. Stream logs in real time:

kubectl logs -f <pod-name>

3. Filter logs:

kubectl logs <pod-name> | grep "ERROR"

4. View logs across all Pods:

kubectl logs -l app=<app-label>

Step 5: Monitoring Alerts with Prometheus

Create an alert for high CPU usage:

1. Rule YAML:

groups:
- name: example-alerts
  rules:
  - alert: HighCPUUsage
    expr: node_cpu_seconds_total > 0.8
    for: 1m
    labels:
      severity: warning
    annotations:
      summary: "High CPU usage detected"

2. Apply the alert:

kubectl apply -f prometheus-alert-rule.yaml

3. Verify:

kubectl get prometheusrule -n monitoring

Best Practices for Monitoring and Logging

1. Set Alerts: Use Prometheus alerts to catch issues proactively.
2. Enable Centralized Logging: Collect and analyze logs using the EFK stack.
3. Monitor Resources: Regularly check kubectl top nodes and kubectl top pods.
4. Retain Logs Securely: Encrypt and back up logs to meet compliance requirements.
5. Visualize Metrics: Use Grafana dashboards for better insights.

Production Example: Comprehensive Monitoring and Logging

1. Deploy a production application with RBAC-enabled Prometheus monitoring.
2. Centralize logs using the EFK stack.
3. Configure Grafana dashboards for visualizing application performance.
4. Set up Prometheus alerts for resource usage thresholds.

Conclusion

In this chapter, you learned:

1. How to monitor Kubernetes clusters using Prometheus and Metrics Server.
2. How to set up centralized logging with the EFK stack.
3. How to troubleshoot issues using logs and metrics.