Since 2015, Confluent (the company behind Kafka) is bringing innovative enhancements to Kafka. These new features include Kafka Connect Framework, Kafka Streams API, Exactly Once Semantic, Kafka SQL etc.

Confluent Kafka distribution is much more ahead than Cloudera Kafka distribution, therefore, we are elaborating steps to install Confluent Kafka on Cloudera distribution.

About Cloudera Hadoop Distribution

Cloudera is the market trend in Hadoop space and is the first one to release commercial Hadoop distribution. It offers to consulting services to bridge the gap between – “what does Apache Hadoop provides” and “what organizations need”.

Cloudera Distribution is:

• Fast for business: From analytics to data science and everything in between, Cloudera delivers the performance you need to unlock the potential of unlimited data.
• Makes Hadoop easy to manage: With Cloudera Manager, automated wizards let you quickly deploy your cluster, irrespective of the scale or deployment environment.
• Secure without compromise: Meets stringent data security and compliance needs without sacrificing business agility. Cloudera provides an integrated approach to data security and governance.

About Confluent Kafka

Confluent created an open source event streaming platform and reimagined it as an enterprise solution. Streaming data as events enables completely new ways of solving problems at scale.

         ●  Founded in 2014 by the creators of Kafka

         ●  Includes many committers to the Apache Kafka project

         ●  Provides support, consulting, and training for Kafka and its ecosystem

         ●  Develops Confluent Community Edition

         ●  Kafka with additional components

         ●  Completely free, open source

Installation Steps

Note:
Java 8 version is required
Use below command to check the version.

$ java -version

If Java 8 version is not installed you can install using below steps
Step 1: Remove the JDK 1.7 version because you need JDK 1.8 version

$ sudo rm -rf /usr/java

Step 2: Install jdk 1.8 rpm
https://www.dropbox.com/s/as5sjhy09kwsznv/jdk-8u131-linux-x64.rpm?dl=0

$ rpm -ivh <jdk>

Step 3: Open sudo and comment and add –

$ sudo gedit /etc/profile

export JAVA_HOME=/usr/java/jdk1.8.0_131

Step 4: Reboot the system

$ sudo reboot

Check the version of maven and java

$ java -version

Step 5: Stop the Zookeeper Service First

$ sudo service zookeeper-server stop

Install Confluent
Step 6: Install the Confluent public key, which is used to sign packages in the YUM repository.

$ sudo rpm --import https://packages.confluent.io/rpm/4.1/archive.key

Step 7: Add the repository to your /etc/yum.repos.d/ directory in a file named confluent.repo

[Confluent]
name=Confluent repository
baseurl=https://packages.confluent.io/rpm/4.1
gpgcheck=1
gpgkey=https://packages.confluent.io/rpm/4.1/archive.key
enabled=1

Recommended: Clear YUM caches

$ sudo yum clean all

The repository is now ready for use.
Step 8: Install Confluent Platform.

$ sudo yum install confluent-platform-2.11

Step 9: Run this command to start all Confluent Platform services by using the CLI.

$ confluent start

Congratulations, you are now up and running with Confluent Kafka on Cloudera Distribution of Hadoop. Now you can build end to end solutions to cater to real world problems involving both platforms!!