JJPP: JP in JP

Writing stuff so I don't forget what I've learned

Pulsar with Manager and Dashboard on Docker Compose

Note that some/most of these are just copied verbatim from their corresponding github READMEs. Also pardon the formatting problems, I was using Notion to write my notes and I just copy-pasted all of this from the markdown. As a side-note, Notion is a very useful tool once you get comfortable with working with Markdown.

Deploying a cluster on Docker

Specifically using docker-compose

A docker-compose file provided by the official git of Pulsar more or less does what we want it to do.

However this looks either old or not really updated (which is the same thing), not the least because pulsar-dashboard is now deprecated and replaced by pulsar-manager.

I have gathered some up-to-date information from Stack, official documentations and my own trial and error. Here is a working docker-compose file:

version: "3.5"
services:
  pulsar:
    image: "apachepulsar/pulsar:2.6.2"
    command: bin/pulsar standalone
    environment:
      PULSAR_MEM: " -Xms512m -Xmx512m -XX:MaxDirectMemorySize=1g"
    volumes:
      - ./pulsar/data:/pulsar/data
    ports:
      - "6650:6650"
      - "8080:8080"
    restart: unless-stopped
    networks:
      - network_test_bed

  pulsar-manager:
    image: "apachepulsar/pulsar-manager:v0.2.0"
    ports:
      - "9527:9527"
      - "7750:7750"
    depends_on:
      - pulsar
    environment:
      SPRING_CONFIGURATION_FILE: /pulsar-manager/pulsar-manager/application.properties
    networks:
      - network_test_bed

networks:
  network_test_bed:
    name: network_test_bed
    driver: bridge

Simple Example

Producer

# Producer Dockerfile
FROM python:3.8.6
WORKDIR /usr/src/app
COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt
COPY ..
CMD [ "python", "./producer.py" ]

# Requirements.txt
pulsar-client

# producer.py
import pulsar
client = pulsar.Client('pulsar://pulsar:6650')
producer = client.create_producer(topic='in',
                                  send_timeout_millis=10000)
producer.send(("10,20").encode('utf-8'))
client.close()

Run all of this with both a Pulsar and consumer running (subscribed to the topic “in”) and it should function correctly.

cd [Dockerfile Location]
docker build -t my-python-app .
docker run -it --rm --network="host" my-python-app

Consumer

The Dockerfile will be similar to the Producer one, the only difference is on the code itself.

import pulsar

client = pulsar.Client('pulsar://pulsar:6650')
consumer = client.subscribe('my-topic',
                            subscription_name='my-sub')

while True:
    msg = consumer.receive()
    print("Received message: '%s'" % msg.data())
    consumer.acknowledge(msg)

client.close()

Accessing the Pulsar-Manager

If you started Pulsar Manager using docker or docker-compose, the Pulsar Manager is running at port 9527. You can access the Pulsar Manager UI at http://127.0.0.1/#/environments.

You could access it from: http://localhost:9527

You could create a super user with the following command, to access the dashboard.

CSRF_TOKEN=$(curl http://localhost:7750/pulsar-manager/csrf-token)
curl \
    -H "X-XSRF-TOKEN: $CSRF_TOKEN" \
    -H "Cookie: XSRF-TOKEN=$CSRF_TOKEN;" \
    -H 'Content-Type: application/json' \
    -X PUT http://localhost:7750/pulsar-manager/users/superuser \
    -d '{"name": "admin", "password": "apachepulsar", "description": "test", "email": "username@test.org"}'

localhost: The IP address or domain name of the backend service, in our case, localhost

Creating an environment

An environment represents a Pulsar instance or a group of clusters you want to manage. A Pulsar Manager is capable of managing multiple environments.

  • Click “New Environment” button to add an environment.
  • Input the “Environment Name”. The environment name is used for identifying an environment.
  • Input the “Service URL”. The Service URL is the admin service url of your Pulsar cluster.
    • You need to make sure the service url that Pulsar Manager is able to access. In this example, both pulsar container and pulsar-manager container are linked. So you can use pulsar container name as the domain name of the pulsar standalone cluster. Thus you can type http://pulsar:8080. Note that pulsar here corresponds to the container name you used for the broker.

Features

With pulsar-manager you can manage the following:

  • Tenants Management
  • Namespaces Management
  • Topics Management
  • Subscriptions Management
  • Brokers Management
  • Clusters Management
  • Dynamic environments with multiple changes
  • Support JWT Auth

Apache Pulsar Grafana Dashboard

This part of the setup expects the pulsar-standalone to be working already.

Prometheus

Generate a Prometheus config file.

Install j2cli. j2cli is a command-line tool for templating Jinja2 template files. You can use j2cli to generate a Prometheus config file from the standalone template.

Get a copy of the template yaml and store in ./prometheus

curl -O -J https://github.com/streamnative/apache-pulsar-grafana-dashboard/blob/master/prometheus/standalone.yml.template
pip install j2cli[yaml]
STANDALONE_HOST="$(ifconfig | grep "inet " | grep -v 127.0.0.1 | awk '{ print $2 }')" j2 prometheus/standalone.yml.template > /tmp/standalone.prometheus.yml

Your new configuration file will be populated as follows, for me the STANDALONE_HOST is 163.221.68.230

---
global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.
  evaluation_interval: 15s # By default, scrape targets every 15 seconds.
  # scrape_timeout is set to the global default (10s).
  external_labels:
    cluster: 'standalone'

# Load and evaluate rules in these files every 'evaluation_interval' seconds.
# rule_files:

scrape_configs:

  # - job_name: "proxy"
  #   honor_labels: true # don't overwrite job & instance labels
  #   static_configs:
  #   - targets:
  #     - '163.221.68.230:8080'

  - job_name: "broker"
    honor_labels: true # don't overwrite job & instance labels
    static_configs:
    - targets:
      - '163.221.68.230:8080'

  - job_name: "bookie"
    honor_labels: true # don't overwrite job & instance labels
    static_configs:
    - targets:
      - '163.221.68.230:8080'

  - job_name: "zookeeper"
    honor_labels: true
    static_configs:
    - targets:
      - '163.221.68.230:8080'

I commented out the proxy because i was not sure if i have them in the stand alone, probably had.

Proxy Metrics: This renders the metrics related to Pulsar proxies if you have run proxies in your Pulsar clusters. This doesn’t apply to a standalone cluster.

Pulsar-Manager github

Grafana

When you have a Pulsar standalone and a Prometheus server connecting to the Pulsar standalone, you can start with the Grafana Dashboard. Replace the values in the docker-compose file with the ones below:

PULSAR_PROMETHEUS_URL=http://$(ifconfig | grep "inet " | grep -v 127.0.0.1 | awk '{ print $2 }'):9090
PULSAR_CLUSTER=standalone

grafana:
    image: streamnative/apache-pulsar-grafana-dashboard:0.0.14
    environment:
      PULSAR_CLUSTER: PULSAR_CLUSTER
      PULSAR_PROMETHEUS_URL: PULSAR_PROMETHEUS_URL
    restart: unless-stopped
    ports:
      - "3000:3000"
    networks:
      - network_test_bed
    depends_on:
      - prometheus

Access the Grafana Dashboard at http://localhost:3000. The default user name is admin, the default password is happypulsaring.

Final Docker compose

docker-compose-servers.yaml 

version: "3.5"
services:
  pulsar:
    image: "apachepulsar/pulsar:2.6.2"
    command: bin/pulsar standalone
    environment:
      PULSAR_MEM: " -Xms512m -Xmx512m -XX:MaxDirectMemorySize=1g"
    volumes:
      - ./pulsar/data:/pulsar/data
    ports:
      - "6650:6650"
      - "8080:8080"
    restart: unless-stopped
    networks:
      - network_test_bed

  pulsar-manager:
    image: "apachepulsar/pulsar-manager:v0.2.0"
    ports:
      - "9527:9527"
      - "7750:7750"
    depends_on:
      - pulsar
    environment:
      SPRING_CONFIGURATION_FILE: /pulsar-manager/pulsar-manager/application.properties
    networks:
      - network_test_bed

  redis:
    image: "redislabs/redistimeseries:1.4.7"
    ports:
      - "6379:6379"
    volumes:
      - ./redis/redis-data:/var/lib/redis
    environment:
      - REDIS_REPLICATION_MODE=master
      - PYTHONUNBUFFERED=1
    networks: 
      - network_test_bed

  alertmanager:
    image: prom/alertmanager:v0.21.0
    ports:
      - "9093:9093"
    volumes:
      - ./alertmanager/:/etc/alertmanager/
    networks:
      - network_test_bed
    restart: always
    command:
      - '--config.file=/etc/alertmanager/config.yml'
      - '--storage.path=/alertmanager'

  prometheus:
    image: prom/prometheus:v2.23.0
    volumes:
      - ./prometheus/standalone.prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"
    networks:
      - network_test_bed

  grafana:
    image: streamnative/apache-pulsar-grafana-dashboard:0.0.14
    environment:
      PULSAR_CLUSTER: "standalone"
      PULSAR_PROMETHEUS_URL: "http://163.221.68.230:9090"
    restart: unless-stopped
    ports:
      - "3000:3000"
    networks:
      - network_test_bed
    depends_on:
      - prometheus


networks:
  network_test_bed:
    name: network_test_bed
    driver: bridge

Certain containers are added and used by me personally. But they all have a hand in the creation of the “middleware”.

References

JJPP

,

One response to “Pulsar with Manager and Dashboard on Docker Compose”

Leave a comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.