spark-mapwithstate Why does the number of storage spaces remain 20?

I am using spark mapwithstate, but the storage space continues to grow.

Question 1.

MapPartitionsRDD Size in Memory 9GB x 20 Can you reduce this size?

Question 2.

And in InternalMapWithStateDStream, storagelevel is fixed to MEMORY_ONLY.


I want to change to persist (StorageLevel.MEMORY_ONLY_SER) because of its size. Is it possible?

Question 3.

Private [streaming] object InternalMapWithStateDStream {
Private val DEFAULT_CHECKPOINT_DURATION_MULTIPLIER = 10
}


I want to reduce this checkpoint value. Is it possible?

2 Answers
2

MapPartitionsRDD Size in Memory 9GB x 20 Can you reduce this size?

It's 9GB x 20, but how is it distributed across your cluster? You need to click the RDDs name to see the distributed state. If you want to reduce the in memory size, you'll need to think of an efficient way to represent your data.

I want to change to persist (StorageLevel.MEMORY_ONLY_SER) because of
its size. Is it possible?

No, you can't override the in-memory settings of the MapWithStateDStream.

MapWithStateDStream

I want to reduce this checkpoint value. Is it possible?

Yes, you can set the DStreams checkpoint interval:

DStream

dStream.mapWithState(spec).checkpoint(Seconds(4))


val kafkastateStream = chnlStream.mapWithState(stateSpec) kafkastateStream.checkpoint(Seconds(10 * 5)) val kafkaSnapshotStream = kafkastateStream.stateSnapshots()

MapWithState also caches mappedValues, so you can also reduce the batch interval to reduce the size of cached RDDs.

MapWithState

mappedValues

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.