n <- 100000 val x = Math.random(); with the livy.server.port config option). def sample(p):

sum(val) Here’s a step-by-step example of interacting with Livy in Python with the x, y = random.random(), random.random() It is strongly recommended to configure Spark …

Pi. } piFunc <- function(elem) { If the Livy service goes down after you've submitted a job remotely to a Spark cluster, the job continues to run in the background. More interesting is using Spark to estimate

We’ll start off with a Spark … val <- ifelse((rands1^2 + rands2^2) < 1, 1.0, 0.0) 2.0. rdd <- parallelize(sc, 1:n, slices) Livy uses the Spark configuration under SPARK_HOME by default. if (x*x + y*y < 1) 1 else 0
NUM_SAMPLES = 100000 When Livy …
We’ll start off with a Spark session that takes Scala code: Once the session has completed starting up, it transitions to the idle state: Now we can execute Scala by passing in a simple JSON command: If a statement takes longer than a few milliseconds to execute, Livy returns You can override the Spark configuration by setting the SPARK_CONF_DIR environment variable before starting Livy. Livy provides high-availability for Spark jobs running on the cluster. print "Pi is roughly, """ cat("Pi is roughly", 4.0 * count / n, ", Apache License, Version By default Livy runs on port 8998 (which can be changed with the livy.server.port config option). }.reduce(_ + _); import random val <- ifelse((rands[1]^2 + rands[2]^2) < 1, 1.0, 0.0) rands2 <- runif(n = length(elems), min = -1, max = 1) By default Livy runs on port 8998 (which can be changed return 1 if x*x + y*y < 1 else 0 println(, """

val NUM_SAMPLES = 100000; rands1 <- runif(n = length(elems), min = -1, max = 1) This is from the Spark Examples: PySpark has the same API, just with a different initial request: The Pi example from before then can be run as: """ piFuncVec <- function(elems) { count <- reduce(lapplyPartition(rdd, piFuncVec), sum) Here’s a step-by-step example of interacting with Livy in Python with the Requests library. rands <- runif(n = 2, min = -1, max = 1) } early and provides a statement URL that can be polled until it is complete: That was a pretty simple example. message(length(elems)) Here is a couple of examples. Requests library. val y = Math.random(); val count = sc.parallelize(1 to NUM_SAMPLES).map { i => count = sc.parallelize(xrange(0, NUM_SAMPLES)).map(sample).reduce(lambda a, b: a + b) val