hadoop developer

Test Pass Academy has Expert Instructors that have been doing Hadoop Developer training since its first came out. Our instructors are well known in the industry not only as top-level instructors with rave reviews, but also as top-level Professionals who pass along real-world examples to the class. Our experts have a vast understanding of Hadoop, Big Data, Hive, Spark and a remarkable teaching ability making it easy to LEARN Hadoop. If you are looking to gain Hadoop Developer Experience, you found the right place. This bootcamp is geared for Corporations, Government, Military and Contractors.

4-Day Hadoop Developer Bootcamp: Train-Test-Pass!

Upon completion of the Hadoop Developer Course, you will demonstrate competence and master:

1. Introduction to Hadoop
2. HDFS
3. YARN
4. Data Ingestion
5. HBase
6. Oozie
7. Working with Hive
8. Hive advanced
9. Hive in Cloudera/Hortonworks Distribution (or tools of choice)
10. Working with Spark
11. Spark Basics
12. Spark Shell
13. RDDs
14. Spark Dataframes and Datasets
15. Spark SQL
16. Spark API programming
17. Spark and Hadoop
18. Machine Learning (ML/MLlib)
19. GraphX
20. Spark Streaming

Hadoop Developer Outline:

Note: As this is an emerging technology course, the outline will be updated as needed

Introduction to Hadoop

• Hadoop history, concepts
• Ecosystem
• Distributions
• High-level architecture
• Hadoop myths
• Hadoop challenges
• Hardware and software

HDFS

• Design and architecture
• Concepts (horizontal scaling, replication, data locality, and rack awareness)
• Daemons: Namenode, Secondary Namenode, and Datanode
• Communications and heart-beats
• Data integrity
• Read and write path
• Namenode High Availability (HA), Federation

YARN

• YARN Concepts and architecture
• Evolution from MapReduce to YARN
• Data Ingestion
• Flume for logs and other data ingestion into HDFS
• Sqoop for importing from SQL databases to HDFS, as well as exporting back to SQL
• Copying data between clusters (distcp)
• Using S3 as complementary to HDFS
• Data ingestion best practices and architectures
• Oozie for scheduling events on Hadoop

HBase

• Concepts and architecture
• HBase vs RDBMS vs Cassandra
• HBase Java API
• Time series data on HBase
• Schema design

Oozie

• Introduction to Oozie
• Features of Oozie
• Oozie Workflow
• Creating a MapReduce Workflow
• Start, End, and Error Nodes
• Parallel Fork and Join Nodes
• Workflow Jobs Lifecycle
• Workflow Notifications
• Workflow Manager
• Creating and Running a Workflow
• Oozie Coordinator Sub-groups
• Oozie Coordinator Components, Variables, and Parameters

Working with Hive

• Architecture and design
• Data types
• SQL support in Hive
• Creating Hive tables and querying
• Partitions
• Joins
• Text processing
• Labs: various labs on processing data with Hive

Hive advanced

• Transformation and Aggregation
• Working with Dates, Timestamps, and Arrays
• Converting Strings to Date, Time, and Numbers
• Create new Attributes, Mathematical Calculations, and Windowing Functions
• Use Character and String Functions
• Binning and Smoothing
• Processing JSON Data
• Execution Engines (Tez, MR, Spark)

Hive in Cloudera or HortonWorks Distribution (or tools of choice)

• Impala architecture
• Impala joins and other SQL specifics

Spark Basics

• Big Data, Hadoop, and Spark
• What’s new in Spark v2
• Spark concepts and architecture
• Spark ecosystem (core, spark sql, mlib, and streaming)

Spark Shell

• Spark web UIs
• Analyzing dataset

RDDs

• RDDs concepts
• RDD Operations/transformations
• Labs: Unstructured data analytics using RDDs
• Data model concepts
• Partitions
• Distributed processing
• Failure handling
• Caching and persistence
• Spark Dataframes and Datasets
• Intro to Dataframe/Dataset
• Programming in Dataframe/Dataset API
• Loading structured data using Dataframes

Spark SQL

• Spark SQL concepts and overview
• Defining tables and importing datasets
• Querying data using SQL
• Handling various storage formats: JSON/Parquet/ORC

Spark API programming (Scala and Python)

• Introduction to Spark API
• Submitting the first program to Spark
• Debugging/logging
• Configuration properties

Spark and Hadoop

• Hadoop Primer: HDFS/YARN
• Hadoop + Spark architecture
• Running Spark on YARN
• Processing HDFS files using Spark
• Spark and Hive

Machine Learning (ML/MLlib)

• Machine Learning primer
• Machine Learning in Spark: MLlib/ML
• Spark ML overview (newer Spark2 version)
• Algorithms: Clustering, Classifications, and Recommendations

GraphX

• GraphX library overview
• GraphX APIs

Spark Streaming

• Streaming concepts
• Evaluating Streaming platforms
• Spark streaming library overview
• Streaming operations
• Sliding window operations
• Structured Streaming
• Continuous streaming
• Spark and Kafka streaming