Introduction to Scala¶

Feng Li¶

Central University of Finance and Economics¶

feng.li@cufe.edu.cn ¶

Course home page: https://feng.li/distcomp ¶

Spark programming in Scala or Python?¶

Spark is written in Scala as it can be quite fast because it's statically typed and it compiles in a known way to the JVM.
Though Spark has API’s for Scala, Python, Java and R but the popularly used languages are the former two.
Java does not support Read-Evaluate-Print-Loop(REPL). R is not a general purpose language.
The data science community is divided in two camps;
- one which prefers Scala
- whereas the other preferring Python.
Each has its pros and cons and the final choice should depend on the outcome application.

Performance¶

Scala is frequently over 10 times faster than Python.
Scala uses Java Virtual Machine (JVM) during runtime which gives is some speed over Python in most cases.
Python is dynamically typed and this reduces the speed.
Compiled languages are faster than interpreted.
In case of Python, Spark libraries are called which require a lot of code processing and hence slower performance.
Scala works well for limited cores.
Moreover Scala is native for Hadoop as its based on JVM.
Scala interacts with Hadoop via native Hadoop's API in Java. It's very easy to write native Hadoop applications in Scala.

Learning Curve¶

Both are functional and object oriented languages which have similar syntax in addition to a thriving support communities.
Scala may be a bit more complex to learn in comparison to Python due to its high-level functional features.
Python is preferable for simple intuitive logic. Python has simple syntax and good standard libraries in the data science community.
Scala is more useful for complex workflows.

Concurrency¶

Scala has multiple standard libraries and cores which allows quick integration of the databases in Big Data ecosystems.
Scala allows writing of code with multiple concurrency primitives.
Python has limited support concurrency or multithreading.
Scala allows better memory management and data processing.
Technically, Python does support heavyweight process forking but only one thread is active at a time. So whenever a new code is deployed, more processes must be restarted which increases the memory overhead.

Usability¶

Both are expressive and we can achieve high functionality level with them.
Python is more user friendly and concise.
Scala is always more powerful in terms of framework, libraries, implicit, macros etc.
Scala works well within the MapReduce framework because of its functional nature.
Many Scala data frameworks follow similar abstract data types that are consistent with Scala’s collection of APIs.
Developers just need to learn the basic standard collections, which allow them to easily get acquainted with other libraries.

Some advices¶

Spark is written in Scala so knowing Scala will let you understand and modify what Spark does internally.
Moreover many upcoming features will first have their APIs in Scala and Java and the Python APIs evolve in the later versions.
But for NLP, Python is preferred as Scala doesn’t have many tools for machine learning or NLP.
Moreover for using standard Spark library, like MLLib, Python is preferred for the data science community.
Python’s visualization libraries complement Pyspark as neither Spark nor Scala have anything comparable.

A bit of background¶

Scala was created by Martin Odersky, who studied under Niklaus Wirth, who created Pascal and several other languages.
Mr. Odersky is one of the co-designers of Generic Java, and is also known as the “father” of the javac compiler.

Overview of Scala¶

Before we jump into the examples, here are a few important things to know about Scala:

It’s a high-level language
It’s statically typed
Its syntax is concise but still readable — we call it expressive
It supports the object-oriented programming (OOP) paradigm -- Every variable is an object, and every “operator” is a method.
It supports the functional programming (FP) paradigm -- Functions are also variables, and you can pass them into other functions. You can write your code using OOP, FP, or combine them in a hybrid style.
It has a sophisticated type inference system
Scala code results in .class files that run on the Java Virtual Machine (JVM)
It’s easy to use Java libraries in Scala

Comments¶

One good thing to know up front is that comments in Scala are just like comments in Java (and many other languages):

// a single line comment

/*
 * a multiline comment
 */

/**
 * also a multiline comment
 */

Create variables:¶

val defines a fixed immutable value — should be preferred.
var creates a mutable variable, and should only be used when there is a specific reason to use it.

In [4]:

val x = 1   //immutable
var y = 0   //mutable

x = 1
y = 0

Out[4]:

In Scala, you typically create variables without declaring their type. When you do this, Scala can usually infer the data type for you. You could check this with Scala REPL or Spark Shell

In [7]:

val x = 1
val s = "a string"

x = 1
s = a string

Out[7]:

a string

This feature is known as type inference, and it’s a great way to help keep your code concise. You can also explicitly declare a variable’s type, but that’s not usually necessary:

In [9]:

val x: Int = 1
val s: String = "a string"

x = 1
s = a string

Out[9]:

a string

Scala build-in data types¶

Scala comes with the standard numeric data types you’d expect. In Scala all of these data types are full-blown objects (not primitive data types).

In [17]:

val b: Byte = 1
val x: Int = 1
val l: Long = 1
val s: Short = 1
val d: Double = 2.0

b = 1
x = 1
l = 1
s = 1
d = 2.0

Out[17]:

2.0

Because Int and Double are the default numeric types, you typically create them without explicitly declaring the data type:

In [18]:

val i = 123   // defaults to Int
val x = 1.0   // defaults to Double

i = 123
x = 1.0

Out[18]:

1.0

For large numbers Scala also includes the types BigInt and BigDecimal.
A great thing about BigInt and BigDecimal is that they support all the operators you’re used to using with numeric types.

In [27]:

var b0 = BigInt(987654321)
var b1 = BigInt(1234567890)
var b2 = BigDecimal(123456.789)

b0 + b1
b0 += 1 
b1 * b1

b0 = 987654322
b1 = 1234567890
b2 = 123456.789

Out[27]:

1524157875019052100

List¶

The List class is a linear, immutable sequence.
All this means is that it’s a linked-list that you can’t modify.
Any time you want to add or remove List elements, you create a new List from an existing List.

In [28]:

val ints = List(1, 2, 3)
val names = List("Joel", "Chris", "Ed")

ints = List(1, 2, 3)
names = List(Joel, Chris, Ed)

Out[28]:

List(Joel, Chris, Ed)

You prepend elements to a List like this:

In [31]:

val b = 0 +: ints

b = List(0, 1, 2, 3)

Out[31]:

List(0, 1, 2, 3)

In [32]:

val b2 = List(-1, 0) ++: ints

b2 = List(-1, 0, 1, 2, 3)

Out[32]:

List(-1, 0, 1, 2, 3)

: character represents the side that the sequence is on.
so when you use +: you know that the list needs to be on the right

In [33]:

val c1 = ints :+ 4

c1 = List(1, 2, 3, 4)

Out[33]:

List(1, 2, 3, 4)

In [36]:

val c2 = ints :+ List(4,5)

c2 = List(1, 2, 3, List(4, 5))

Out[36]:

List(1, 2, 3, List(4, 5))

Control structures¶

if/else Scala’s if/else control structure is similar to other languages:

  if (test1) {
      doA()
  } else if (test2) {
      doB()
  } else if (test3) {
      doC()
  } else {
      doD()
  }

However, unlike Java and many other languages, the if/else construct returns a value, so, among other things, you can use it as a ternary operator:

In [11]:

val a = 5
val b = 7
val x = if (a < b) a else b

a = 5
b = 7
x = 5

Out[11]:

`match` expressions¶

Scala has a match expression, which in its most basic use is like a Java switch statement:

In [13]:

// val i = 1
val i = 5

val result = i match {
    case 1 => "one"
    case 2 => "two"
    case _ => "not 1 or 2"
}

i = 5
result = not 1 or 2

Out[13]:

not 1 or 2

The match expression isn’t limited to just integers, it can be used with any data type, including booleans:

In [15]:

val test = true

val boolean2String = test match {
    case true => "it is true"
    case false => "it is false"
}

test = true
boolean2String = it is true

Out[15]:

it is true

`for` loops and expressions¶

In [17]:

// "x to y" syntax
for (i <- 0 to 5) println(i)

In [18]:

// "x to y by" syntax
for (i <- 0 to 10 by 2) println(i)

You can also add the yield keyword to for-loops to create for-expressions that yield a result. Similar to the list comprehension in Python. Here’s a for-expression that doubles each value in the sequence 1 to 5:

In [19]:

val x = for (i <- 1 to 5) yield i * 2

x = Vector(2, 4, 6, 8, 10)

Out[19]:

Vector(2, 4, 6, 8, 10)

Here’s another for-expression that iterates over a list of strings:

In [21]:

val fruits = List("apple", "banana", "lime", "orange")

val fruitLengths = for {
    f <- fruits
    if f.length > 4
} yield f.length

fruits = List(apple, banana, lime, orange)
fruitLengths = List(5, 6, 6)

Out[21]:

List(5, 6, 6)

`while` and `do/while`¶

Scala also has while and do/while loops. Here’s their general syntax:

    // while loop
    while(condition) {
        statement(a)
        statement(b)
    }

    // do-while
    do {
       statement(a)
       statement(b)
    } 
    while(condition)

Classes¶

Here’s an example of a Scala class:

In [1]:

class Person(var firstName: String, var lastName: String) {
    def printFullName() = println(s"$firstName $lastName")
}

defined class Person

In [8]:

val myname = new Person("Feng", "Li")

myname = Person@7aeeb4fc

Out[8]:

Person@7aeeb4fc