By: Heather Miller, Martin Odersky, and Philipp Haller
Functional programming languages are regularly touted as an enabling force, as an increasing number of applications become concurrent and distributed. However, managing closures in a concurrent or distributed environment, or writing APIs to be used by clients in such an environment, remains considerably precarious-- complicated environments can be captured by these closures, which regularly leads to a whole host of potential hazards across libraries/frameworks in Scala's standard library and its ecosystem.
Potential hazards when using closures incorrectly:
This SIP outlines an abstraction, called spores, which enables safer use of closures in concurrent and distributed environments. This is achieved by controlling the environment which a spore can capture. Using an assignment-on-capture semantics, certain concurrency bugs due to capturing mutable references can be avoided.
In the following example, an Akka actor spawns a future to concurrently process incoming requests.
Example 1:
Improvement Processdef receive = {
case Request(data) =>
future {
val result = transform(data)
sender ! Response(result)
}
}
Capturing sender in the above example is problematic, since it does not return a stable value. It is possible that the future's body is executed at a time when the actor has started processing the next Request message which could be originating from a different actor. As a result, the Response message of the future might be sent to the wrong receiver.
The following example uses Java Serialization to serialize a closure. However, serialization fails with a NotSerializableException due to the unintended capture of a reference to an enclosing object.
Example 2:
case class Helper(name: String)
class Main {
val helper = Helper("the helper")
val fun: Int => Unit = (x: Int) => {
val result = x + " " + helper.toString
println("The result is: " + result)
}
}
Given the above class definitions, serializing the fun member of an instance of Main throws a NotSerializableException. This is unexpected, since fun refers only to serializable objects: x (an Int) and helper (an instance of a case class).
Here is an explanation of why the serialization of fun fails: since helper is a field, it is not actually copied when it is captured by the closure. Instead, when accessing helper its getter is invoked. This can be made explicit by replacing helper.toString by the invocation of its getter, this.helper.toString. Consequently, the fun closure captures this, not just a copy of helper. However, this is a reference to class Main which is not serializable.
The above example is not the only possible situation in which a closure can capture a reference to this or to an enclosing object in an unintended way. Thus, runtime errors when serializing closures are common.
The main idea behind spores is to provide an alternative way to create closure-like objects, in a way where the environment is controlled.
A spore is created as follows.
Example 3:
val s = spore {
val h = helper
(x: Int) => {
val result = x + " " + h.toString
println("The result is: " + result)
}
}
The body of a spore consists of two parts:
In general, a spore { ... } expression has the following shape.
Note that the value declarations described in point 1 above can be implicit but not lazy.
Figure 1:
spore {
val x_1: T_1 = init_1
...
val x_n: T_n = init_n
(p_1: S_1, ..., p_m: S_m) => {
<body>
}
}
The types T_1, ..., T_n can also be inferred.
The closure of a spore has to satisfy the following rule. All free variables of the closure body have to be either
Example 4:
case class Person(name: String, age: Int)
val outer1 = 0
val outer2 = Person("Jim", 35)
val s = spore {
val inner = outer2
(x: Int) => {
s"The result is: ${x + inner.age + outer1}"
}
}
In the above example, the spore's closure is invalid, and would be rejected during compilation. The reason is that the variable outer1 is neither a parameter of the closure nor one of the spore's value declarations (the only value declaration is: val inner = outer2).
The type of the spore is determined by the type and arity of the closure. If the closure has type
A => B, then the spore has type Spore[A, B]. For convenience we also define spore types for two or more parameters.
In example 3, the type of s is Spore[Int, Unit].
Implementation
The spore construct is a macro which
The Spore trait for spores of arity 1 is declared as follows:
trait Spore[-T, +R] extends Function1[T, R]
For each function arity there exists a corresponding Spore trait of the same arity (called Spore2, Spore3, etc.)
An invocation of the spore macro expands the spore's body as follows. Given the general shape of a spore as shown above, the spore macro produces the following code:
new <spore implementation class>[S_1, ..., S_m, R]({
val x_1: T_1 = init_1
...
val x_n: T_n = init_n
(p_1: S_1, ..., p_m: S_m) => {
<body>
}
})
Note that, after checking, the spore macro need not do any further transformation, since implementation details such as unneeded remaining outer references are removed by the new backend intended for inclusion in Scala 2.11. It's also useful to note that in some cases these unwanted outer references are already removed by the existing backend.
The spore implementation classes follow a simple pattern. For example, for arity 1, the implementation class is declared as follows:
class SporeImpl[-T, +R](f: T => R) extends Spore[T, R] {
def apply(x: T): R = f(x)
}
Similar to regular functions and closures, the type of a spore should be inferred. Inferring the type of a spore amounts to inferring the type arguments when instantiating a spore implementation class:
new <spore implementation class>[S_1, ..., S_m, R]({
// ...
})
In the above expression, the type arguments S_1, ..., S_m, and R should be inferred from the expected type.
Our current proposal is to solve this type inference problem in the context of the integration of Java SAM closures into Scala. Given that it is planned to eventually support such closures, and to support type inference for these closures as well, we plan to piggyback on the work done on type inference for SAMs in general to achieve type inference for spores.
We now revisit the motivating examples we described in the above section, this time in the context of spores.
The safety of futures can be improved by requiring the body of a new future to be a nullary spore (a spore with an empty parameter list).
Using spores, example 1 can be re-written as follows:
def receive = {
case Request(data) =>
future(spore {
val from = sender
val d = data
() => {
val result = transform(d)
from ! Response(result)
}
})
}
In this case, the problematic capturing of this is avoided, since the result of this.sender is assigned to the spore's local value from when the spore is created. The spore conformity checking ensures that within the spore's closure, only from and d are used.
Using spores, example 2 can be re-written as follows:
case class Helper(name: String)
class Main {
val helper = Helper("the helper")
val fun: Spore[Int, Unit] = spore {
val h = helper
(x: Int) => {
val result = x + " " + h.toString
println("The result is: " + result)
}
}
}
Similar to example 1, the problematic capturing of this is avoided, since helper has to be assigned to a local value (here, h) so that it can be used inside the spore's closure. As a result, fun can now be serialized without runtime errors, since h refers to a serializable object (a case class instance).
Contents
Copyright © 2011-2013 EPFL. All rights reserved.
