Real-Time Analytics using NiFi and Spark
Categories:
Big Data
Reviews:
The interactive virtual workshop is focussed on Building Data Flows, Performing Streaming Ingestion and Streaming Processing using NiFi and Spark Streaming. This program covers below aspects in NiFi and Spark –
- Fundamentals,
- Limitations,
- Architectures,
- Hands-on Exercises
Spark API perspective below areas will be covered –
- Micro-batch Processing
- Structured Streaming Processing
Hands-on exercises related to Spark will be in Python.
- Application Developers
- DevOps Engineers
- Architects
- System Engineers
- Technical Managers
Introduction to Streaming
- Streaming Overview
- Streaming Use cases
- Streaming Frameworks
- NiFi vs Kafka
- Kafka Streams vs Spark Streaming
- Challenges in Streaming
- Which one to choose when?
Introduction to NiFi
- NiFi Overview
- Limitations of NiFi
- NiFi Terms
- Hands-on: Setting up NiFi
- Demo: Working with NiFi User Interface
NiFi Architecture
- Logical Architecture of NiFi
- Physical Architecture of NiFi
- Roles and Responsibilities of various components
- Hands-on exercise
Working with NiFi Flows
- Anatomy of FlowFile
- Key NiFi Processors
- Configuring Queues
- NiFi Templates
- Hands-on exercise
Delving Deeper in NiFi
- NiFi Registry
- NiFi Expression Language
- Controller Service
- Hands-on exercise
Integrating NiFi with Kafka
- Role of Kafka
- Kafka Terms
- Kafka Processors
- Hands-on exercise
Spark Streaming Overview
- Spark Logical Architecture
- Various APIs of Spark Streamings
- DStream API vs Structured Streaming
- Terms and Concepts
Streaming ETL using Spark
- Processing streaming data using Spark
- Various Types of Windowing
- Watermarking and Triggers Concept
- Stateless Streaming
- Hands-on exercise: Stateless Streaming
- Stateful Streaming
- Hands-on exercise: Stateful Streaming
- Structured Streaming
- Hands-on exercise: Structured Streaming
Integrations and Flows using Streaming Concepts
- Structured Streaming Integrations
- Kafka Key Configurations
- Hands-on: Integrating Kafka with Spark Streaming
- Hands-on Integrate Spark Streaming with HBase
- Hands-on: Build End to End Flow from Twitter to HBase
- Basic Knowledge of Big Data Technologies as covered in Big Data Crash Course or equivalent knowledge
- Basic knowledge of Python
Upon completion of this course, you should be able to:
- Understand and Gain Hands-on exposure regarding
- Key Streaming Frameworks
- Pros and Cons of NiFi, Spark and Kafka
- Spark and NiFi Fundamentals & Architecture,
- Integration of NiFi and Spark Streaming via Apache Kafka
- Spark Micro-batch processing
- Structured Streaming Processing