Spark java.lang.OutOfMemoryError: Java heap space

My cluster: 1 master, 11 slaves, each node has 6 GB memory.

My settings:

spark.executor.memory=4g, Dspark.akka.frameSize=512


Here is the problem:

First, I read some data (2.19 GB) from HDFS to RDD:

val imageBundleRDD = sc.newAPIHadoopFile(...)


Second, do something on this RDD:

val res = imageBundleRDD.map(data => {
val desPoints = threeDReconstruction(data._2, bg)
(data._1, desPoints)
})


Last, output to HDFS:

res.saveAsNewAPIHadoopFile(...)


When I run my program it shows:

.....
14/01/15 21:42:27 INFO cluster.ClusterTaskSetManager: Starting task 1.0:24 as TID 33 on executor 9: Salve7.Hadoop (NODE_LOCAL)
14/01/15 21:42:27 INFO cluster.ClusterTaskSetManager: Serialized task 1.0:24 as 30618515 bytes in 210 ms
14/01/15 21:42:27 INFO cluster.ClusterTaskSetManager: Starting task 1.0:36 as TID 34 on executor 2: Salve11.Hadoop (NODE_LOCAL)
14/01/15 21:42:28 INFO cluster.ClusterTaskSetManager: Serialized task 1.0:36 as 30618515 bytes in 449 ms
14/01/15 21:42:28 INFO cluster.ClusterTaskSetManager: Starting task 1.0:32 as TID 35 on executor 7: Salve4.Hadoop (NODE_LOCAL)
Uncaught error from thread [spark-akka.actor.default-dispatcher-3] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[spark]
java.lang.OutOfMemoryError: Java heap space


There are too many tasks?

PS: Every thing is ok when the input data is about 225 MB.

How can I solve this problem?

7 Answers
7

I have a few suggestions:

spark.executor.memory=6g

spark.storage.memoryFraction

cache()

persist

String

Map

WrappedArray

Dataset

Dataset

http://spark.apache.org/docs/1.2.1/configuration.html

EDIT: (So I can google myself easier) The following is also indicative of this problem:

java.lang.OutOfMemoryError : GC overhead limit exceeded


To add a use case to this that is often not discussed, I will pose a solution when submitting a Spark application via spark-submit in local mode.

Spark

spark-submit

According to the gitbook Mastering Apache Spark by Jacek Laskowski:

You can run Spark in local mode. In this non-distributed single-JVM deployment mode, Spark spawns all the execution components - driver, executor, backend, and master - in the same JVM. This is the only mode where a driver is used for execution.

Thus, if you are experiencing OOM errors with the heap, it suffices to adjust the driver-memory rather than the executor-memory.

OOM

heap

driver-memory

executor-memory

Here is an example:

spark-1.6.1/bin/spark-submit
--class "MyClass"
--driver-memory 12g
--master local[*]
target/scala-2.10/simple-project_2.10-1.0.jar


Have a look at the start up scripts a Java heap size is set there, it looks like you're not setting this before running Spark worker.

# Set SPARK_MEM if it isn't already set since we also use it for this process
SPARK_MEM=${SPARK_MEM:-512m}
export SPARK_MEM

# Set JAVA_OPTS to be able to load native libraries and to set heap size
JAVA_OPTS="$OUR_JAVA_OPTS"
JAVA_OPTS="$JAVA_OPTS -Djava.library.path=$SPARK_LIBRARY_PATH"
JAVA_OPTS="$JAVA_OPTS -Xms$SPARK_MEM -Xmx$SPARK_MEM"


You can find the documentation to deploy scripts here.

You should increase the driver memory. In your $SPARK_HOME/conf folder you should find the file spark-defaults.conf, edit and set the spark.driver.memory 4000m depending on the memory on your master, I think.
This is what fixed the issue for me and everything runs smoothly

spark-defaults.conf

spark.driver.memory 4000m

The location to set the memory heap size (at least in spark-1.0.0) is in conf/spark-env.
The relevant variables are SPARK_EXECUTOR_MEMORY & SPARK_DRIVER_MEMORY.
More docs are in the deployment guide

SPARK_EXECUTOR_MEMORY

SPARK_DRIVER_MEMORY

Also, don't forget to copy the configuration file to all the slave nodes.

SPARK_EXECUTOR_MEMORY

SPARK_DRIVER_MEMORY

SPARK_EXECUTOR_MEMORY

SPARK_DRIVER_MEMORY

You should configure offHeap memory settings as shown below:

`val spark = SparkSession
.builder()
.master("local[*]")
.config("spark.executor.memory", "70g")
.config("spark.driver.memory", "50g")
.config("spark.memory.offHeap.enabled",true)
.config("spark.memory.offHeap.size","16g")
.appName("sampleCodeForReference")
.getOrCreate()`


Give the driver memory and executor memory as per your machines RAM availability. You can increase the offHeap size if you are still facing the OutofMemory issue.

Broadly speaking, spark Executor JVM memory can be divided into two parts. Spark memory and User memory. This is controlled by property spark.memory.fraction - the value is between 0 and 1.
When working with images or doing memory intensive processing in spark applications, consider decreasing the spark.memory.fraction. This will make more memory available to your application work. Spark can spill, so it will still work with less memory share.

spark.memory.fraction

spark.memory.fraction

The second part of the problem is division of work. If possible, partition your data into smaller chunks. Smaller data possibly needs less memory. But if that is not possible, you are sacrifice compute for memory. Typically a single executor will be running multiple cores. Total memory of executors must be enough to handle memory requirements of all concurrent tasks. If increasing executor memory is not a option, you can decrease the cores per executor so that each task gets more memory to work with.
Test with 1 core executors which have largest possible memory you can give and then keep increasing cores until you find the best core count.

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.