Setting up Confluent Kafka on Cloudera Distribution 5.xx
Since 2015, Confluent (the company behind Kafka) is bringing innovative enhancements to Kafka. These new features include Kafka Connect Framework, Kafka Streams API, Exactly Once Semantic, Kafka SQL etc.
Confluent Kafka distribution is much more ahead than Cloudera Kafka distribution, therefore, we are elaborating steps to install Confluent Kafka on Cloudera distribution.
About Cloudera Hadoop Distribution
Cloudera is the market trend in Hadoop space and is the first one to release commercial Hadoop distribution. It offers to consulting services to bridge the gap between – “what does Apache Hadoop provides” and “what organizations need”.
Cloudera Distribution is:
- Fast for business: From analytics to data science and everything in between, Cloudera delivers the performance you need to unlock the potential of unlimited data.
- Makes Hadoop easy to manage: With Cloudera Manager, automated wizards let you quickly deploy your cluster, irrespective of the scale or deployment environment.
- Secure without compromise: Meets stringent data security and compliance needs without sacrificing business agility. Cloudera provides an integrated approach to data security and governance.
About Confluent Kafka
Confluent created an open source event streaming platform and reimagined it as an enterprise solution. Streaming data as events enables completely new ways of solving problems at scale.
● Founded in 2014 by the creators of Kafka
● Includes many committers to the Apache Kafka project
● Provides support, consulting, and training for Kafka and its ecosystem
● Develops Confluent Community Edition
● Kafka with additional components
● Completely free, open source
Java 8 version is required
Use below command to check the version.
$ java -version
If Java 8 version is not installed you can install using below steps
Step 1: Remove the JDK 1.7 version because you need JDK 1.8 version
$ sudo rm -rf /usr/java
Step 2: Install jdk 1.8 rpm
$ rpm -ivh <jdk>
Step 3: Open sudo and comment and add –
$ sudo gedit /etc/profile
Step 4: Reboot the system
$ sudo reboot
Check the version of maven and java
$ java -version
Step 5: Stop the Zookeeper Service First
$ sudo service zookeeper-server stop
Step 6: Install the Confluent public key, which is used to sign packages in the YUM repository.
$ sudo rpm --import https://packages.confluent.io/rpm/4.1/archive.key
Step 7: Add the repository to your /etc/yum.repos.d/ directory in a file named confluent.repo
[Confluent] name=Confluent repository baseurl=https://packages.confluent.io/rpm/4.1 gpgcheck=1 gpgkey=https://packages.confluent.io/rpm/4.1/archive.key enabled=1
Recommended: Clear YUM caches
$ sudo yum clean all
The repository is now ready for use.
Step 8: Install Confluent Platform.
$ sudo yum install confluent-platform-2.11
Step 9: Run this command to start all Confluent Platform services by using the CLI.
$ confluent start
Congratulations, you are now up and running with Confluent Kafka on Cloudera Distribution of Hadoop. Now you can build end to end solutions to cater to real world problems involving both platforms!!