Scalable Machine Learning on Big Data using Apache Spark

Apache Spark and parallel data processing

1. In the ApacheSpark worker nodes

2. On the local Driver machine

3. sc.parallelize(range(9999999999)).count()

4. ObjectStorage & NoSQL

5. ObjectStorage & NoSQL

6. SQL/Relational Databases

7. Catalyst and Tungsten are able to optimise the execution, so are more likely to execute more quickly than if you would had implemented something equivalent using the RDD API.

The API is simpler and doesn't require specific functional programming skills

Parallelism in Apache Spark 

Question 1

What is the advantage of median over mean?

Median is more outlier resistent. Odd values influence median less than mean.

Question 2

What is the mean of the following list?

1,2,4,5,34,1,32,4,34,2,1,3

Please use a decimal point instead of a comma

10.25

Question 3

What is the median of the following list?

1,2,4,5,34,1,32,4,34,2,1,3

Please use a decimal point instead of a comma

3.5

Question 4

Which of the following two plots has a higher standard deviation?

Plot 2

Question 5

What is the standard deviation of the following list?

34,1,23,4,3,3,12,4,3,1

Please enter at least 3 digits after the decimal

Please use a decimal point instead of a comma

10.5629

Question 6

Which of the two plots indicates a higher kurtosis value?

Plot 1

Question 7

What is the kurtosis of the following list?

34,1,23,4,3,3,12,4,3,1

Please enter at least three digits after the decimal

3.2968

Question 8

What is true about this value distribution?

This distribution is positively skewed

Question 9

What is the correlation between the two lists?

1,2,3,4,5,6,7,8,9,10

7,6,5,4,5,6,7,8,9,10

Please enter at least three digits after the decimal

0.7093

Question 10

What is the covariance between the two lists?

1,2,3,4,5,6,7,8,9,10

7,6,5,4,5,6,7,8,9,10

Please enter at least three digits after the decimal

3.653

Question 11

The correlation between the following two lists is zero, can you explain why?

1,2,3,4,5,6,7

7,6,5,4,5,6,7

Correlation of 1st half of the list is negative and between the last half of the list is positive so they cancel out

SparkML concepts 

Question 1

What are Machine Learning Pipelines?

Please choose all correct answers

A way to do feature engineering within the pipeline framework

A way of speeding up ML development

A way of expressing your complete end-2-end machine learning flow within a single framework with a homogeneous API

Question 2

How is the class called which transforms a string class label to a class index in SparkML?

StringIndexer

Question 3

What is the class called which transforms a set of columns in a data frame to a single 

DenseVector representation in SparkML?

VectorAssembler

Course Project Quiz

1. Eh Number pettina Wrong osthadhi

2. 5.30775

3. 5.11174

4. 0.730365

5. 0.8

6. 0.8

7. 0.8

8. 0.2666

9. CSV, with header, columns separated by comma

10. Spearman & Pearson

11. 3

12. 2