Testing Leader Election (in Raft) using Akka, without Thread.sleep() ;-)

Posted by Konrad 'ktoso' Malawski under coding, english, fun, scala (No Respond)

As I’m currently living in London and miss the Kraków SCKRK Reading Club meetings a lot – (they’re basically the best thing since sliced bread!). Even though I do I attend them remotely anyway nowadays, I still miss having proper discussions on science papers. The same feeling struck me and  some friends (Andrzej from GeeCON, to name one) decided to start a similar initiative in London. It’s called Paper Cup, the Reading Club, and is a “reading club”, which means that we select Computer Science Whitepapers to read, read them at home, and then during a meeting discuss them fiercly :-)

For this week’s meeting we decided to read Raft (Raft – In Search of an Understandable Consensus Algorithm), which is “very trendy” right now (and relatively fresh; the paper is from Oct 2013!). But to sum it up quickly – it’s an algorithm trying to solve the same problem as Paxos (achieving consensus in distributed systems), yet being understandable at the same time. We did read Paxos more than a few times (the Lamport’s “Paxos Made Simple” paper for example), but it was always a challenge to get everything right with it. Raft’s authors on the other hand really wanted the algorithm to be understandable, as well as still “good enough” performance wise.

As the algorithm is really so nice, and described in terms of a simple State Machine, I figured I’d implement it using Akka (and FSM) for the next meeting, and another friend is implementing it on Clojure. To be fair, it seems I’m not the first person to have figured out that Akka’s FSM and Actors in general will make this super fun to implement – there are already some repos doing exactly that (well, some of them being “initial commit” ;-)). In this post I won’t be focusing on the algorithm itself, but rather, would like to share a useful technique when you implement a thing like this, and would like to write a “big test”, that asserts from a high point of view that the system performs as you’d expect.

The test case we’ll talk about is very simple:

  1. Raft should elect a leader right away after the nodes start
  2. Raft should re-elect a leader if we kill the current one

As I already said, we won’t be diving into the algorithm here – if you’re curious you can just read the whitepaper (and maybe even drop by for PaperCup? :-)).
Let’s jump right into the tests:

Assume we’ve already spun up a few actors  performing the Raft algorithm (talking with each other in order to find a leader) before executing these tests. The setup itself is not so interesting, and you can check the full test “base” file on github if you’re interested. Now the real question as with all async algorithms is “how long do I wait for completion?“. The answer in our case is “Until someone becomes Leader.“ So we want to be fishing for such a transition, or message stating this change in the raft cluster.

Each member is running an instance of Raft, and one of them (by election) will after some time become the Leader. One way to test this would be to plan a TestProbe in the “members to notify”, or use the probe as intermediate between members (A -> Probe -> B). But that’s a lot of work to be honest. The solution I’m proposing here, is also used by akka-persistence (well, and my akka-persistence-hbase plugin to it too) – let’s call it “Event Stream Subscription Awaiting“. Now that we have a fancy name for it, let’s continue with implementing it.

Let’s start from the end of the story. I want to block until some event occurs in the system (the awaitElectedLeader() method does this). Where can I fish for those events? Turns out Akka has a built in eventBus ready to messages into (and it’s available for every ActorSystem, without any additional setup). Let’s first implement our helper methods, for awaiting on an ElectedAsLeader message:

Here (in the a test class extending TestKit) we can get access to the EventStream, and subscribe for specific types of messages on it. We’ll be using a TestProbe() to recieve these events, because it allows us to expectMsg* which is exactly what we need. So in awaitElectedLeader() we just await using the probe, until the message of “some Leader has appeared in the cluster” comes in. So far this is not so different what I described before and then called it “a lot of work”. Now we’ll get to the actual trick in this method though. The eventStream you see here is defined on the ActorSystem, and as we know, that is also accessible from within an Actor itself.

In akka-persistence for example Actors publish events about the progress of the replay / confirmations etc. But that is only for testing, so you can enable a flag (persistence.publish-confirmations for example) to make it publish events to the eventStream. That’s a great idea and makes testing very simple, and would be certainly possible to implement in my Raft implementation (and probably will end up like this anyway). Let’s now however think how we could extend a receive of an Actor, to automatically send incoming messages also to the EventStream.

It’s very simple actually, if you know about Actor’s aroundReceive method. Just like it’s name implies (sounds very much like AOP by the way, doesn’t it? ;-)), it allows you to wrap a receive call of an actor with your code. Our implementation will simply just send all messages to the eventStream:

Easy! Now we don’t want to change our production code (or maybe it’s not our code, etc etc), so during Actor creation we can use a type refinement to mix this trait into our RaftActor (during setup, that I briefly mentioned, but didn’t show explicitly):

And this Actor will now automatically send all messages to the EventStream. On the event stream we subscribe for LeaderElected events using our probe, and the expectMsg part we already talked about. So… that’s it! A very simple method to test actors “from the outside”. Now you can look back at the first code-snippet in this post, to see how this all fits together.

All in all, if you’re building anything using Actors, or maybe even a library someone might want to test, it’s a great idea to provide a setting to enable your Actors sending events the EventStream, because then even from the outside, it’s easy to track what’s happening deep down in the system. I’m pretty sure I will include such “enableable” events for the libraries I’m currently building – in order to be nice to the lib’s users. ;-)

Tags: , , , , , , ,

NoJarTool – Run Scalding jobs on remote cluster from SBT

Posted by Konrad 'ktoso' Malawski under big data, coding, english, scala, twitter (No Respond)

I’ve just released a small but very helpful class which enables you to run Scalding (Hadoop) Jobs on a remote cluster, without the need of creating a so-called fat-jar you’d pass in to hadoop jar [jar]. This saves me on average 10 minutes of dev time before I can start a job on my cluster, and is extremely useful during development – when I run against small datasets.

Using the Tool is rather simple, as it only wraps around an existing Tool, in this case the scalding.Tool:

This runner will include all files located in the target/scala-2.10/classes directory as well as the all-deps.jar which should contain all your dependencies (such as Scalding, Cascading, and all other 3rd party libraries) in the “distributed cache” (by using the -libjars option, which in source is actually called “tmpjars” but that’s another story… ;-)), which is sent over to all nodes executing the job. The huge gain here is that the “dependency-jar” so you don’t need to run assembly each time you want to run a job on the cluster (which can take quite some time…).

As for creating the “dependencies jar”, I’d recommend using you can use the sbt assembly plugin and it’s great assemblyPackageDependency command. It might also be a good idea to automate the “build new jar when dependencies have changed” step – for that you can use appendContentHash feature which my pal from SoftwareMill, Krzysztof Ciesielski developed => sbt assembly content hashing.

The idea came from a discussion on cascading-user, where debs (including me) were a bit unsure why Cascading sometimes decided to use LocalJobRunner instead of the real one.

For more info, and sources of the Tool, check ktoso/hadoop-scalding-nojartool.

PS: Oh man, how come I didm’t blog anything else Scalding related before? Been playing around with it over a year now… Oh well :-)

Tags: , , , , , ,

Disrupt 2 Grow — “Life is Study!”

Posted by Konrad 'ktoso' Malawski under freedom (1 Respond)

Life is Study! - Kintaro Oe (Golden Boy)

Since the videos from this year’s Devoxx are online already, I figured I’d post my quicky session that was titled “Disrupt 2 Grow”.
The talk was very unusual for me to do, as I’d usually only talk about “the code“, but once I started preparing this talk which is a bit “soft and fluffy”, I discovered there’s quite a bit to tell about my last few years. I think this talk, though only the tip of the iceberg, is quite crucial to how I’ve been living my life recently, so if you’d like to get into that topic, and perhaps discuss about it, why not start with this video. I already got some fantastic discussion-email about some of the topics to which I can’t wait to properly reply — finally a real educated discussion via email, whoa! Anyhow, the video is only 15 minutes long, so you might give the video a try, here it is:

Following that presentation Tori Weldt (also known as “@java” on twitter) did a short interview with me, about what I’m up to recently, why I advocate Scala recently and how I feel about Kraków‘s and Poland‘s in general communities (they’re the best) – such as JUGs (Polish (Kraków), Warsaw, Poznań, Lublin), the Kraków Lambda Lounge the GDGs (GDG Kraków, Warsaw, Poznań and more…) and of course the Kraków Scala User Group — all groups I either was very active with or even helped founding. And then of course there’s Geecon, which we’re incredibly proud of (hint: call for papers is open). It was a good year when you think about it this way :-)

Devoxx-wise still, I also participated in facilitating the BOF about the “Devoxx for Kids” initiative, where I talked about how we did our Kraków edition of that idea – Geecon 4 kids. All in all, I think it was good to come back to Devoxx this year – it’s always good to meet old friends, and that’s something Devoxx is good at – because of the sheer amount of developers attending.

PS: Merry x-mas!

Tags: , , , , , ,

Scala reflection basics – accessing an object’s private field explained

Posted by Konrad 'ktoso' Malawski under english, scala (No Respond)

Today’s blog post will be on the slightly weird side, because as Scala developers we don’t usually need to reach into reflection to solve problems at hand. It’s easy to fight most problems with the “there’s a monad for that!” tactic, or simply “adding yet another layer solves any software development problem!”.

Sadly today’s task is not easily solvable by just that. The problem we’re going to tackle using Scala’s reflection involves a scala object and a private field we need to access.

Problem statement: We need to support parallel integration tests (sbt running tests in parallel), using Mongo (or Fongo). The MongoDB object though, contains globaly shared mutable state, in form of the dbs Map, which is used to determine which MongoAddress to hit for which entity.

We are able to setup the mongo identifiers up properly for the threads executing the tests, so they don’t interfere with each other, and we won’t focus on the setup part today. Instead let’s focus on what happens during teardown. A naive implementation is a simple (ScalaTest) afterAll, like this:

So sadly the naive implementation with using MongoDB.close() is not enough for us. Why? Imagine Two threads, running two MongoFlatSpecs, assume each has it’s own in memory Fongo instance even. There still is that shared MongoDB.dbs field deep in lift-mongodb’s internals. If we’d use MongoDB.close() we’ll clean all MongoIdentifiers, so we might break tests which are still in flight… We could try something among the lines of “wait for the right moment to clean up”, but … why wait!? There’s a ConcurrentHashMap in there, and we just want to remove “the one MongoIdentifier I have been using for this MongoFlatSpec”.

*Reflection to the rescue!* So we’ll have to use reflection, in order to execute such statement: MongoDB.dbs.remove(myMongoIdentifier). The first problem is that we’re dealing with a Scala object, also known as “Module”. Why is it also known as *Module*? Let’s take a look into the ByteCode!

Since we want the dbs field, we will have to go through the static final MongoDB$ MODULE$ field of the MongoDB$ class. While it’s certainly doable using plain Java reflection, let’s see how Scala’s (still experimental) reflection can make this a bit nicer:

The first thing to notice here is the use of Mirrors instead of explicitly calling getDeclaredField and it’s friends on Class<?> objects like you would in plain Java APIs. Why the hassle? The grand idea behind Mirrors (other than “it sounds so cool – reflection => mirrors, yay!“) is that you can obtain either a runtime or macros universe. This is discussed in detail in the environment-universes-mirrors section of the Scala docs if you want to dig deeper. The quick overview though is this: when you think about it, AST and reflection stuff are pretty similar. I mean, we traverse the same things, only at a different time (compile-time vs. runtime), so some things may be not available in the runtime, but why change the entire API if we could find some common ground – that is, Mirrors. Sure, they will be a bit different if used from the macros universe than from the runtime universe, but as we also write compile time macros the benefit is huge – only one API with slight flavoring to learn, instead of two separate ones – depending on the stage of when we apply them.

Coming back to our code though, as you can see, we were able to avoid going through the magic MODULE$ constant explicitly, it was abstracted away from us thanks to the types of mirrors we’ve been using – it’s also worth noting that for example the reflectModule(ModuleSymbol) takes a ModuleSymbol which is obtained via the “safe cast”, as one might think of it in the line above using the asModule method. The nice thing about asModule is that it will fail if the term you’re calling it on is not a module. So the API is both designed to be as typesafe as possible, even in reflection-lang, as well as “fail fast” if it detects you’re doing something wrong.

For more docs or further reference check out the docs about scala-reflection or ScalaDoc about scala.reflect.api.Mirrors.

Tags: , , , ,

Java8 The Good Parts @ JavaOne 2013

Posted by Konrad 'ktoso' Malawski under coding (2 Responds)

Just yesterday I gave a presentation together with Andrzej Grzesik, with whom we (and more friends!) organize GeeCON and the PolishJUG among other things.

This is our first time at JavaOne, and right away we got a full room of developers, eager to learn about the new stuff in JDK8. I have to admit, being a Scala guy since quite a few years, presenting about Java 8 felt a bit weird, but I’m still very happy about it – especially the JVM changes which allowed Java 8 and it’s Lambdas to happen. All these changes will not only make Java developer lives easier (less-painful), but also allow our Scala code to become loads faster! :-)

The slides from the talk are online here:

Oh, and it seems that if you link to http://oracle.com/technetwork/java/, which is the OTN website, you get a free t shirt, yay!

Tags: , , , , , , , , , ,

Proof of Concept – LambdaSpec, how testing will look with Java8

Posted by Konrad 'ktoso' Malawski under coding, english, java (No Respond)

I recently had an hour or two to spare, so I decided to play around with Java 8 and it’s Lambdas. Coming from Scala and knowing the Spec of lambdas (I’ve been checking it out on each update) I went straight on to implementing anything useful with them.

I decided that Java will eventually get a testing framework using lambdas, in a more BDD way. My favourite example of such framework is ScalaTest (by Bill Venners) which i wholeheartedly love nowadays. So my goal was to implement something similiar, wihin what’s possible with Java currently.

And here’s how a passing / failing and pending test would look like:

So pretty much like you’d expect it from a good testing framework nowadays. Truth be told – reading junit in the terminal is a horrible pain so you have to use your IDE for browsing test failures. (I honestly like my terminal output to be meaningful enough for testing – that way I always have all tests running on one screen, and am writing code on the main screen).

You may ask why this verbosity is a nice thing. That is: “Why not use annotations, like we always have?”. One of the answers is the bellow snippet, which displays how generating test data, and meaningful test names, and the tests themselfes are very easy to track and understand (this is not yet implemented):

Or another style (yet to be improved):

LambdaSpec is for now just a Proof of Concept, but eventually I hope to get it to the point that once J8 is released, we’ll be able to use it instead of JUnit. Some of the nice features I want to bring in would be smart test dependency analysis (so only the tests that may have failed will be run) and a separate runner… Think ~test if you know ScalaTest… :-)

Anyway, if you’re interested in Java 8 and what’s coming up, i encourage you to star the project (it’ll give me the necessary power to push this thing forward ;-)).

Check out the sources at https://github.com/ktoso/lambda-spec

Tags: , , , , ,

[presentation][polish] TDD as the way to Scala enlightenment

Posted by Konrad 'ktoso' Malawski under coding, polish (1 Respond)

Yesterday I was presenting at the local Student IT Festival together with a friend – Tomasz Borek. We’re both quite active community people as well as TDD fans thus when I was asked to present at SFI again this year – an idea came to mind. Let’s teach Tom Scala, guiding him via writing tests.

The target audience was fresh students who have yet to learn the ways of TDD (“be enlightened” ;-)), so while the things we cover in this presentation are pretty basic for those of us hacking in the industry since some years – I believe it’s tremendously useful for people to get into “real world programming”.

The presentation is already online (although in polish):

And there’s also an video recoding of the talk online already. The SFI team will probably upload their recording soon enough – which should be in higher quality than this ‘homemade’ one ;-)

[SFI 2013] TDD drogą do oświecenia w Scali – Konrad Malawski, Tomasz Borek from Polish Java User Group on Vimeo.

Tags: , , , , , , ,

Scala 2.10 – Macros hands on with “method alias”

Posted by Konrad 'ktoso' Malawski under coding, english, scala (1 Respond)

Where?

Last week I was presenting at JFokus (cool conference, give it a try), a really nice JVM-focused conference. Obviously I love the conference I co-organise with fellow JUG members Geecon, too but it’s always fantastic to be a participant/speaker rather than the organiser once in a while – then you actually can enjoy the talks and not run around arranging everything… :-)

After the last day of the conference there was a Stockholm Scala User group meetup which I’ve obviously had to go to :-) With a beer in hand we jokingly invented a @alias annotation that would allow you to setup aliases for symbolic methods. An example would be akka’s (scala api) that delegates to tell (java api). While I had nothing to do waiting for the plane to arrive at the airport, I hacked up a proof of concept macro that does this.

As a heads up, all the code is on github: https://github.com/ktoso/scala-macro-method-alias

What?

I came up with two syntaxes, the first one being more scala-like:

And another one, wich looks more java-like (due to the annotation ;-)), and we’ll in the joke we said it should be an annotation:

Also it has been a great occasion to try out how implementing macros actually feels – and I gotta say, we’re not in lisp heaven here, but it’s pretty ok once you get a feel for it. Static typing for the win I’d say. Also, in 2.11 we’ll probably get a “quasiquoting string interpolator”, which would make creating Trees even easier.

In this post we’ll look at how implementing a macro in Scala 2.10 looks like. We’ll just cover the 1st syntax mentioned here (1st image).

How?

Let’s start by looking at the code then, shall we? On the call site we don’t have much to do. We just import the macro, and call it with the method (partially applied). The macro will then look at this method’s signature and delegate all params to it.

Very well… Let’s move on to the implementation. It has to be an separately compilable object. The compiler will compile it, and call the macros implementation while it’s compiling the class in which our call site is. In this aliasFor method, we take the partially applied call, and “invoke” the macro (notice the macro keyword).

Now it’ll get a bit more interesting. Let’s start with looking at line 1 – the macro’s def .
It’s closely related to the invocation of the macro, see the delegate parameter here? It must be named the same way as the delegate parameter in aliasFor. If it isn’t or the signature here has more/less parameters than the signature of aliasFor – you’ll get a compile error, informing you that those must match.

Still in the signature, let’s look at c: Context, where’d that come from? You probably know – it’s the AST of our “surroundings”. You can call c.enclosingMethod or c.enclosingClass to get Tree‘s back from it. Also, it contains c.universe which may be thought of like “THE Scala entry point to reflection-land”, but since we’re still in compile-time, we get a compile-time universe. Using it we can create new AST nodes, or inspect anything that’s reachable from our code. It’s a big topic to wrap your head around, so it’s best to see the docs about it.

Next we’ll use this c to obtain what the method name of the one calling us is, and we can parse delegate to obtain the name of the method we want to delegate to. It will actually have the type information associated with it there by the way. Next we extract parameters that the delegator method has, and in the end, parse a small piece of scala code which basically is just the invocation of the target method + all the parameters it needs.

Notice that if someone would pass in more than one partially applied method into our aliasFor method, we would go into the _ case in here, and abort execution. The c.abort method causes the compiler to stop, and print out the position where it failed, along with the message we supply (there’s an example failure on the github of this project).

Lastly, having a Tree we need to convert it to an Expr – as that’s what a macro should return. Exprs are bound to an universe, but since in compile time, there’s only one universe – we can go ahead with using this simple c.Expr factory (it wouldn’t be that easy in runtime).

Sources?

That’s it… a quick overview on how writing macros looks with scala. I definitely recommend reading up about Universes and Def Macros before you head out and rewrite everything to be a macro ;-) As for the use case of this project… Well, in theory it’s useful, but the win is rather small – so I wouldn’t advertise using it everywhere it may fit ;-) It is on maven central if you’re interested to try it out though.

All usage information and sources sources (available under the Apache 2 License) are on GitHub, so check it out:
https://github.com/ktoso/scala-macro-method-alias

Tags: , , , , , , , , ,

Scala 2.10 – or, why you will love implicit value classes

Posted by Konrad 'ktoso' Malawski under coding, java, scala (6 Responds)

Hip hip hurray. Scala 2.10 is Final (as can be read on scala-internals) so it’s high time for another blog post celebrating all the awesomeness it’s going to bring into our daily lifes!

This time we’ll cover a pretty awesome SIP, namely: SIP-15 Value Classes. If you’re not familiar at all with the idea you should either read the SIP now, or stay with me learn why they’re so awesome, and then read the SIP. I’ll give you a paragraph of dots to think about it.

Seems you’re back! Let’s get hacking then.

So a Value Class is a smart way of encapsulating a number (for example) with a class, to make it efficient to use such simple wrapper classes. Let’s look into the “Wins” of Value Classes in two examples

Win 1: Better type checking, near-none performance costs

Before we start implementing stuff, a quick reminder about Scala’s Type Hierarchy:

skitch (5)

Focus on the Any class and it’s two children for now. On the right, we have AnyRef, which is Java’s Object, and all our classes that we write in Scala (or Java) extend it. Scala implementations have the additional layer of ScalaObject, but that’s it – this is the side of the type tree where our objects live.

With Value Classes this changes…

as we’ve become able to write classes which are on the AnyVal side of this hierarchy! This isn’t exactly a new feature for the compiler, as you can see that’s how we’ve been using java “int” which was decorated as “scala.Int” for years now. What changed in 2.10 is the fact that we can now add our own classes on this branch of the tree!

Let’s just see how Value Classes work in this blogpost, and I’ll post a more concrete example in another post, as this one is getting pretty long :-) The first question we need to answer is – “Why being on the left side here is such a big deal?”. Let’s start out with just a Meter (as in Distance) class:

Seems pretty normal. The only difference, which made it a Value Class, is the fact that we explicitly extend AnyVal. Now is a good moment to take a look at the generated bytecode (just type the example in an object Test in the REPL and the run javap -v Test to see for yourself):

Now you should probably “get” what the Value class changed… Yes, the class Meter is not used in the final bytecode! Instead we’re operating on a plain value – the Int parameter that Meter was wrapping. Thus, we didn’t add any memory overhead connected to introducing this class!

Ok, but what if this class will have some methods?

Let’s move the traveled method inside the Meter class and find out:

We’ve now moved the traveled method inside of our Meter class. What does this mean for the runtime – just an int won’t do, since we’re calling things on it… right?

As you can see – an int is plenty enough to support even method calls “on it”! The compiler is smart enough to replace all Meter.traveled calls with a static call to it’s companion object Meter$.traveled(int) which as you can see… again takes a plain int, instead of an Meter instance!

That’s pretty sweet – we the type safety of Meter at compile time, and we can use it to make our methods “feel strongly about” what is passed into them – accept only Meter, yet at runtime we don’t have ANY memory increase due to passing around big objects – we’re back to primitives!

Win 2: Implicit methods get a boost!

This “Win” could also be called “Implicits loose the runtime overhead of creating the Wrapper class each time!” or “Scala gets real Extension Methods!”.

Just a quick reminder on implicits in Scala: We can use implicit conversions to decorate some type, with another type and, seemingly, add methods to it. In Scala 2.9 we’d do it like this:

So there are two new things here in Scala 2.10, one is that we can get rid of the asAwesomeString method, which pretty much was just boilerplate to convert a String into an AwesomeString.

Implicit classes are also new in 2.10

This is done by using an implicit class (which are described in detail in SIP-13), but this is not our focus today. Just assume they generate the boilerplate called “asAwesomeString” for you, and work the same way otherwise.

What’s more interesting for us in this blog post here is that by using both features SIP-13 and SIP-15 we can construct an implicit value class. Let’s take a look at the bytecode generated by both, the implicit class as well as the implicit value class to spot the difference:

The implicit class goes first:

Most notably, take a look at the parts of the bytecode marked with ^^^^ and vvvv. The AwesomeString method (generated for us by the compiler) returned the Wrapper class here, and then on the instance of AwesomeString the .awesome_!() method gets called.Pretty normal implicits use so far. Bear in mind that this wrapper creation is done for each implicit method you call – it’s not cached, even if you chain multiple implicit calls on the same String instance that we just wrapped.

Implicit Value Class

In contrast, this is the bytecode generated for an implicit value class:

Let’s again focus on the bytecode marked with vvvv and ^^^^. This time the wrapper function is still called, but it returns the same string instance that came in (ScalaGurus – why can’t the compiler just skip this invocation and call the extension method right away?).

Then this string is passed right into the awesome_!(String) method defined on the companion object. As you see the difference here is that we pass in the unboxed value to the extension method (yeap, let’s call it this way) whereas before we would call this method on the wrapper object.

Turns out that using “plain extension methods”, via defining implicit value classes has noticeable performance benefits (small but hey, it adds up!). And if you would like to see this used somewhere else than just “some guys blog”, look no futher and check out StringOps in Scala’s sources.

More info @ ScalaWags

Since Daniel, Dick & Josh have just recenly discussed this (and other) features of Scala 2.10 I’d like to link their awesome ScalaWags podcast here. This is an exact link to the part where they discuss value classes as well as implicit classes, so feel free to grab a coffee and hear then out nerding around the Scala Collections library – something we all love to do :-) Oh and be sure to follow @TheScalaWags on twitter.

Shameless Plug: GeeCON 2013 – Call For Papers

A small shameless plug… As we’re right in the middle of preparing 2013′s “All around the JVM” GeeCON conference, I thought that bringing in even more awesome Scala people would be… even more awesome? ;-) So, if you’re interested in anything around the JVM (Scala? :-)) – why not submit your talk to our C4P?

Tags: , , , , , , ,

Use Scala’s DynamicVariable to avoid closing over a value in Akka

Posted by Konrad 'ktoso' Malawski under coding, english, scala (2 Responds)

Note: As Steve points out in the comments this is actually NOT SAFE (the solution with DynamicVariable). (!)

If you’ve ever read Akka’s docs about Actors or saw this blogbost about how to avoid closing over the sender in Akka you already know why you shouldn’t close over the sender value when implementing actors with futures inside them. In this post we’ll discover a nice Scala utility class we can use to solve this problem.

But first… what is the problem?

// BROKEN! DON'T DO THIS AT HOME!
  1. class Caplin extends Actor {
  2.   def recieve = {
  3.     case _ => Future { sender ! "カピバラ" } // OUCH!
  4.   }
  5. }

The above code may fail, as the variable sender (it’s actually a method) is “shared mutable state”! Why? Well, there’s one instance of Caplin, yet the sender changes each time someone sends Caplin a message right? So while the future is executing, the value of sender may have changed (as another message just came in). The above code would then end up responding to the wrong sender.

Now with having the problem defined, let’s see how we can solve it. The obvious solution is to create a copy of the sender before we dispatch the Future, just like:

// this is OK
  1. class Caplin extends Actor {
  2.   def recieve = {
  3.     case _ =>
  4.       val mySender = sender
  5.       Future { mySender ! "カピバラ" }
  6.   }
  7. }

This works OK. But I promised to talk about scala.util.DynamicVariable in the blog post’s title, didn’t I? Let’s see how we could implement the same thing by using it then.

DynamicVariable is pretty much like Java’s ThreadLocal. It exposes a value, which may be set to different values per Thread – so that’s exactly what we need to safe-guard our code agains mixing up the sender values.

// dynamic variable version
  1. class Caplin extends Actor {
  2.   val Sender = new DynamicVariable[ActorRef](self)
  3.  
  4.   def recieve = {
  5.     case "カピバラ" => "Why am I talking with myself?"
  6.     case _         =>
  7.       Sender.withValue(sender) {
  8.         Future { Sender.value ! "カピバラ" }
  9.       }
  10.   } 
  11. }

As you can see, first we create the “thread-local” value Sender and before we dispatch the new Future we wrap the code with DynamicVariable::withValue which assures that within this block of code – the value of sender is guaranteed to be what we passed in there.

Where does the name DynamicVariable come from you might ask? It’s clearly not dynamic in the “Type” way – it’s strictly typed as you’d expect it to be. Well, turns out that what we see here is an implementation of Dynamic Scoping (read up about it on Wikipedia).

That’s it for now, check back later to see some more Scala in action, as I’m just in the middle of preparing my Scala + Android talk (version 2!). While this example maybe not the best case (it’s quite a bit of code compared to “just use a val”) on where to use DynamicVariable, I hope more of you are now aware of the close over sender issue, as well as this nice nifty class from Scala.

Tags: , , ,