{"id":7682,"date":"2024-07-03T19:35:06","date_gmt":"2024-07-04T02:35:06","guid":{"rendered":"https:\/\/ioflood.com\/blog\/?p=7682"},"modified":"2024-07-03T19:35:06","modified_gmt":"2024-07-04T02:35:06","slug":"install-kafka-linux","status":"publish","type":"post","link":"https:\/\/ioflood.com\/blog\/install-kafka-linux\/","title":{"rendered":"How to Install Kafka | Kafka Quick Start Guide for Linux"},"content":{"rendered":"
\n
\"Developer<\/figure>\n<\/div>\n

Handling real-time data streams are vital for our automated process at at IOFLOOD<\/a>. While discovering improved tools, kafka CLI stood out as a specialized command-line tool for managing Apache Kafka and real-time data. We have translated our practices into this tutorial for our bare metal cloud server<\/a> customers looking for help within the Apache Kafka getting started phases.<\/p>\n

In this tutorial, we will guide you through Kafka install steps for Linux systems.<\/strong> We will show you methods for both APT and YUM-based distributions, delve into compiling Kafka cli from source, installing a specific version, and finally, how to use the Kafka command and ensure it’s installed correctly.<\/p>\n

So, let’s dive in and begin our Linux Kafka quick start process!<\/p>\n

TL;DR: How To Install Kafka CLI on Linux?<\/h2>\n

\n You can install Kafka by first downloading the Kafka binaries from the Apache website<\/a> with wget https:\/\/downloads.apache.org\/kafka\/3.7.1\/kafka_2.13-3.7.1.tgz<\/code>. Then, extract the tar file with tar xzf kafka_2.13-3.7.1.tgz<\/code>, configure, and start the Kafka server.\n<\/p><\/blockquote>\n

Here’s a basic example:<\/p>\n

wget https:\/\/downloads.apache.org\/kafka\/3.7.1\/kafka_2.13-3.7.1.tgz\n\ntar xzf kafka_2.13-3.7.1.tgz\n\ncd kafka_2.13-3.7.1\n\nbin\/zookeeper-server-start.sh config\/zookeeper.properties &\n\nbin\/kafka-server-start.sh config\/server.properties &\n<\/code><\/pre>\n
This is just a basic way to install Kafka on Linux, but there’s much more to learn about installing and using Kafka. Continue reading for more detailed information and advanced usage scenarios.<\/p>\n
The Basics: Install Kafka on Linux<\/h2>\n
Kafka is an open-source distributed streaming platform by Apache. It’s used to build real-time data pipelines and streaming apps. It’s horizontally scalable, fault-tolerant, and incredibly fast. This makes Kafka a go-to solution for a large variety of real-time data streaming needs.<\/p>\n
Now, let’s dive into how to install Kafka on your Linux system.<\/p>\n
Kafka Install with APT<\/h3>\n
For Debian-based distributions, we use the APT package manager. Here’s how you can install Kafka using APT:<\/p>\n
sudo apt-get update\nsudo apt-get install kafka\n\n# Output:\n# 'kafka is already the newest version (2.13-2.8.0).'\n<\/code><\/pre>\n
This command updates your package lists and then installs Kafka. The output message confirms that Kafka has been installed successfully.<\/p>\n
Kafka Install with YUM<\/h3>\n
For Red Hat-based distributions, we use the YUM package manager. Here’s how you can install Kafka using YUM:<\/p>\n
sudo yum update\nsudo yum install kafka\n\n# Output:\n# 'Package kafka-2.13_2.8.0.noarch already installed and latest version'\n<\/code><\/pre>\n
Similar to the APT commands, these commands update your package lists and then install Kafka. The output message confirms that Kafka has been installed successfully.<\/p>\n
Remember, these are basic installations. For more advanced scenarios, such as installing from source or installing different versions, continue reading to the Advanced Use section.<\/p>\n
Source Code Kafka Quick Start<\/h2>\n
Sometimes, you may need to install Kafka from the source code. This could be due to the need for a specific version or to customize the installation.<\/p>\n
Here’s how you can install Kafka from the source code:<\/p>\n
git clone https:\/\/github.com\/apache\/kafka.git\n\ncd kafka\n\ngradle\n\n# Output:\n# 'BUILD SUCCESSFUL in 2m 36s'\n<\/code><\/pre>\n
This command clones the Kafka repository, changes the directory to the Kafka folder, and then builds Kafka using Gradle. The output message confirms that Kafka has been built successfully.<\/p>\n
Different Versions of Kafka CLI<\/h2>\n
Kafka has numerous versions, and you might need a specific one for compatibility or to use a particular feature. Here’s how you can install a specific version of Kafka.<\/p>\n
Installing from Source<\/h3>\n
To install a specific version from source, you need to checkout the specific version before building with Gradle:<\/p>\n
git clone https:\/\/github.com\/apache\/kafka.git\ncd kafka\ngit checkout 2.3.0\ngradle\n\n# Output:\n# 'BUILD SUCCESSFUL in 2m 36s'\n<\/code><\/pre>\n
Installing with APT<\/h3>\n
To install a specific version with APT, you need to specify the version during installation:<\/p>\n
sudo apt-get update\nsudo apt-get install kafka=2.3.0\n\n# Output:\n# 'kafka is already the newest version (2.3.0).'\n<\/code><\/pre>\n
Installing with YUM<\/h3>\n
To install a specific version with YUM, you need to specify the version during installation:<\/p>\n
sudo yum update\nsudo yum install kafka-2.3.0\n\n# Output:\n# 'Package kafka-2.3.0.noarch already installed and latest version'\n<\/code><\/pre>\n\n\n\n\n\n\n\n
Version<\/th>\nKey Changes<\/th>\nCompatibility<\/th>\n<\/tr>\n<\/thead>\n
2.3.0<\/td>\nAdded support for Java 11<\/td>\nCompatible with all previous versions<\/td>\n<\/tr>\n
2.4.0<\/td>\nIntroduced Incremental Cooperative Rebalancing<\/td>\nCompatible with all previous versions<\/td>\n<\/tr>\n
2.5.0<\/td>\nAdded support for non-key joining in KTable<\/td>\nNot backwards compatible with versions below 2.3.0<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
Use and Verify New Kafka Install<\/h2>\n
Once Kafka is installed, you can use the kafka-topics<\/code> command to create a topic and then list all topics to verify that Kafka is working correctly.<\/p>\n
Here’s an example:<\/p>\n
kafka-topics --create --topic test --partitions 1 --replication-factor 1 --bootstrap-server localhost:9092\nkafka-topics --list --bootstrap-server localhost:9092\n\n# Output:\n# 'test'\n<\/code><\/pre>\n
This command creates a topic named ‘test’ and then lists all topics. The output shows the ‘test’ topic, confirming that Kafka is installed and working correctly.<\/p>\n
How to Install Kafka in Docker<\/h2>\n
Docker is a popular platform used to develop, ship, and run applications. By installing Kafka using Docker, you can ensure that the software will run the same, regardless of its environment.<\/p>\n
Here’s how to install Kafka using Docker:<\/p>\n
docker pull confluentinc\/cp-kafka\n\ndocker run -d --name=kafka -p 9092:9092 -e KAFKA_ZOOKEEPER_CONNECT=your.zookeeper.connect:2181 -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT:\/\/your.kafka.host.name:9092 -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 confluentinc\/cp-kafka\n\n# Output:\n# 'Successfully tagged confluentinc\/cp-kafka:latest'\n<\/code><\/pre>\n
This command pulls the Kafka image and then runs a new Kafka container. The output message confirms that the Kafka Docker image has been successfully created.<\/p>\n
Install Kafka with Kubernetes<\/h2>\n
Kubernetes is an open-source platform designed to automate deploying, scaling, and operating application containers. Here’s how to install Kafka using Kubernetes:<\/p>\n
helm repo add confluentinc https:\/\/confluentinc.github.io\/cp-helm-charts\/\nhelm repo update\nhelm install my-kafka confluentinc\/cp-helm-charts\n\n# Output:\n# 'NAME: my-kafka'\n# 'STATUS: deployed'\n<\/code><\/pre>\n
This command adds the Confluent Helm chart repository, updates the repo, and then installs Kafka using Helm. The output message confirms that Kafka has been successfully deployed on Kubernetes.<\/p>\n\n\n\n\n\n\n
Method<\/th>\nAdvantages<\/th>\nDisadvantages<\/th>\n<\/tr>\n<\/thead>\n
Docker<\/td>\nSimplifies configuration, Isolation from other applications, Easy version control<\/td>\nRequires Docker knowledge, More resources usage<\/td>\n<\/tr>\n
Kubernetes<\/td>\nScalability, High Availability, Automated rollouts and rollbacks<\/td>\nMore complex, Requires Kubernetes knowledge<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
While these alternative methods might seem more complex, they offer greater control and scalability, especially in a microservices environment.<\/p>\n
Solving Kafka Install Issues<\/h2>\n
While installing Kafka on Linux, you may encounter a few common issues. Let’s discuss these problems and their solutions to ensure a smooth Kafka installation process.<\/p>\n
Issue: Kafka Service Not Starting<\/h3>\n
Sometimes, after installation, the Kafka service might not start. This can be due to various reasons such as incorrect configuration or port conflicts.<\/p>\n
To check if Kafka is running, you can use the following command:<\/p>\n
systemctl status kafka\n\n# Output:\n# 'kafka.service - Apache Kafka\n# Loaded: loaded (\/usr\/lib\/systemd\/system\/kafka.service; disabled; vendor preset: disabled)\n# Active: inactive (dead)'\n<\/code><\/pre>\n
If Kafka is not running, you can start it using the following command:<\/p>\n
systemctl start kafka\n\n# Output:\n# No output on success\n<\/code><\/pre>\n
Issue: Kafka Version Conflict<\/h3>\n
If you have installed Kafka from the source and there is a version already installed through the package manager, it can lead to a version conflict. To resolve this, you need to uninstall the existing version before installing a new one.<\/p>\n
Here’s how to uninstall Kafka:<\/p>\n
sudo apt-get remove kafka\n\n# Output:\n# 'The following packages will be REMOVED:\n#   kafka'\n<\/code><\/pre>\n
Issue: Kafka Port Conflict<\/h3>\n
Kafka uses port 9092 by default. If another service is using this port, Kafka will not start. You can check the port usage using the following command:<\/p>\n
sudo lsof -i:9092\n\n# Output:\n# 'COMMAND  PID USER   FD   TYPE DEVICE SIZE\/OFF NODE NAME\n# java    2034 kafka  152u  IPv6  21560      0t0  TCP *:9092 (LISTEN)'\n<\/code><\/pre>\n
If the port is in use, you need to stop the service using the port or change the Kafka port.<\/p>\n
Remember, troubleshooting is a normal part of any installation process. Don’t get discouraged if you encounter issues. With these tips, you should be able to resolve common problems and get your Kafka service running on Linux.<\/p>\n
Understanding What is Kafka<\/h2>\n
Before delving into the installation process of Kafka on Linux, it’s essential to understand what Kafka is and its underlying architecture. This will not only make the installation process more meaningful but also enhance your ability to troubleshoot any potential issues.<\/p>\n
What is Kafka?<\/h3>\n
Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.<\/p>\n
# Kafka version check\nkafka-topics --version\n\n# Output:\n# '2.8.0'\n<\/code><\/pre>\n
The above command checks the installed Kafka version. The output shows the version of the Kafka installed on your system.<\/p>\n
Kafka Architecture<\/h3>\n
The architecture of Kafka is made up of a few key components: Producers, Consumers, Brokers, Topics, Partitions, and Clusters.<\/p>\n