Project Background

“DataCouch Power Ltd.” (fictitious energy utility company) has installed Smart meters at customers’ homes/premises across London, UK. These meters emit energy consumption at small interval of time. Such smart meter readings are sent to the utility company, for them to charge based on how much electrical energy a customer consumes.

In order to optimize cost of operation and provide better Customer experience, this utility company wants to determine overuse of sanctioned load, perform proactive maintenance like identifying faulty meters, plan service outages etc. so that there is minimal impact to Customers.

Apart from above use cases, energy disaggregation service is available from third party that can analyze the energy smart meter logs and give deeper insights about the appliances and other settings that consumes the energy. Such disaggregated data can be used to cross sell information like customer lifestyle, customer profile, appliances under use and their usage pattern to third party.

Project Objectives

• Analyze batch data and provide insights about energy usage pattern in last couple of years.
• Identify Faulty Meters which require attention. This data should be captured in streaming mode.
• Find out average energy consumption by each consumer in every 10 seconds time frame.
• Segregate meters which are consuming energy over and above their sanctioned load.
• Display Dashboard for business to perform Cube Analytics on data.

This project will provide exposure to participants to build Big Data Analytics end to end data pipeline from scratch.

This intensive training course encompasses lectures and hands-on labs that help participants learn theoretical knowledge and gain practical experience of Spark Core, SQL and Streaming. Hands-on exercises will enable participants to work with various common data sources like HDFS, MySQL, HBase, Kafka etc. Also during the course, participants will get exposure to deal with variety of Dataformats including CSV, JSON, XML, Log files, Avro, Parquet, ORC etc. using Spark Framework.

• Learn basics of Scala
• Understand Spark Core, SQL and Streaming Architecture and APIs
• Get Practical exposure to key projects like Spark, Kafka etc.
• Develop Distributed code using the Scala programming language
• Optimize Spark jobs through partitioning, caching, and other techniques
• Build, deploy, and run Spark code on Hadoop clusters
• Transform structured data using SparkSQL and DataFrames
• Process and Analyze Streaming Data using Spark
• Integrate Spark with Kafka, HBase etc.
• Work with key dataformats like Avro, Parquet etc.