Consuming Events

For this exercise we will be supplying two different methods you can consume events from

Kafka:

1. kcat

Use Kcat to connect to our Streaming APIs, for the purpose of these exercises. Kcat

(formerly kafkacat) is a command-line utility that you can use to test and debug

Apache Kafka deployments. You can use kcat to produce, consume, and list topic and

partition information for Kafka.

Follow the kcat installation here. To start consuming events, you would kcat

commands in the following format:

kcat -b -X security.protocol=ssl -t

-C -o end

Which would look similar to this:

kcat -b -X security.protocol=ssl -t

vault.api.v1.customers.customer.created -C -o end

The kafka_broker can be found in the Developer Starter Pack. You can refer to this

kcat documentation for more information.

2. KafkaConsumer in Python (Filtering allowed)

A Python file is provided that allows you to consume from a specified kafka topic and

server. Enter the Kafka topic in the KafkaTopic variable and the Kafka Server in the

KafkaServer Variable.

The following prerequisite installations are required:

Classification | Confidential 4 ©2023 Thought Machine

● Python 3

● Python-snappy module (pip3 install python-snappy)

● Kafka-Python (pip3 install kafka-python)

To set up, navigate to api_tutorials directory and run the following command:

pip3 install -r requirements.txt

This simple Python script accepts five arguments:

● [topic] e.g. vault.core_api.v1.accounts.account.events

● [server] e.g.

bootstrap.kafka.partner-shared-sandbox.tmachine.io:443

● [security_protocol] SSL or Plaintext

● [headers] True or leave empty/false

● [filter] e.g. chris

Example: python kafka-consume-filter.py

vault.core_api.v1.accounts.account.events

bootstrap.kafka.partner-shared-sandbox.tmachine.io:443 Plaintext