Spark performance reduced with increased number of worker nodes

Multi tool use
Multi tool use


Spark performance reduced with increased number of worker nodes



I have a spark application which loads data from csv files, calls Drools engine, uses flatmap and saves results to output files (.csv)



Below are the two test cases :



1) When I am running this Application with 2 Worker nodes having same configuration (8 cores) Application takes 5.2 minutes to complete :



No. Of Executors : 133



Total No of Cores : 16



Used Memory : 2GB( 1GB per executor)



Available Memory : 30GB



2) When I am running this Application with 3 Worker nodes having same configuration (8 cores) Application takes 7.6 minutes to complete :



Expected result



It should take less time after adding one more worker node with same configuration.



Actual Result



It takes more time after adding one more worker node with same configuration.



I am running application using spark-submit command in standalone mode.


spark-submit



Here I want to understand that why increasing worker node doesn't increase the performance, is that not the correct expectation?



EDIT



After looking at other similar question on stackoverflow I tried running application with spark.dynamicAllocation.enabled=true, however it further degrades the performance.


spark.dynamicAllocation.enabled=true





Spark: Inconsistent performance number in scaling number of cores
– user8371915
Jul 3 at 10:37





@user8371915 Thanks for pointing out, however problem in that question is having same machine and increased number of cores(that's why in solution he mentioned multithreading on single machine), in my case machines are multiple(worker nodes are 2 in first case and 3 in second case) , is that solution still remain applicable in my case?
– Raj
Jul 3 at 12:06










By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

2CT4NSVtD,h 5tTw31v9 Rvu NfA
-
8OHGXYnr CeZciaIus3dZD6EwI5cpTiXTdtXd8JQL3z4HTZrBYnHQo8C92vTO,OedlY gekNVZQAIg5NB A 3wObVu8Rxhf9tw

Popular posts from this blog

PHP contact form sending but not receiving emails

PHP contact form sending but not receiving emails I have created a contact form using PHP to send the data. The form seems to send fine as there are no errors and success message appears, however I am not receving the emails into the designated inbox. I am using PHPMailer as I originally tried just using the php 'mail' command which I now understand is a bit hit and miss. I cannot see why I would not be receving the emails so I would be very grateful for any help that could be given. I'm quite new to PHP so please be patient with me :) <?php use PHPMailerPHPMailerPHPMailer; use PHPMailerPHPMailerException; $msg = ""; if (isset($_POST['submit'])) { require 'phpmailer/src/Exception.php'; require 'phpmailer/src/PHPMailer.php'; require 'phpmailer/src/SMTP.php'; function sendemail ($to, $from, $body) { $mail = new PHPMailer(true); $mail->setFrom($from); $mail->addAddress($to); $ma...
Read more

Do graphics cards have individual ID by which single devices can be distinguished?

Do graphics cards have individual ID by which single devices can be distinguished? I have several graphics cards of the same manufacturer and same model and I want to distinguish them not only by the pcie socket they reside in but by some individual value I can read from the hardware. Is there such value and if so what would it be called and how could I access it? Right now I'm using modern nvidia cards (10xx series) and nvidia inspector.but I'd prefer if there was any value that was also applicable to amd. I have programming and scripting knowledge. I am running Windows 10 1 Answer 1 This is a powershell example to pull information from video cards. The DeviceID is supposed to be unique but only if the manufacturer supports Windows Display Driver Model (WDDM). (I copied this directly from docs.microsoft.com) $strComputer = "." $colItems = get-wmiobject -class "Win32_VideoC...
Read more

Create weekly swift ios local notifications

Create weekly swift ios local notifications I can create daily notifications but not weekly ones using swift 3 ios 10 User Notifications framework. Daily works fine but when I add in the .weekday parameter it doesn't send me a notification. The code is: for i in 1 ... 7 { let center = UNUserNotificationCenter.current(); let content = UNMutableNotificationContent(); content.title = "Title"; content.body = "content"; content.sound = UNNotificationSound.default(); var dateComponents = DateComponents(); dateComponents.weekday = i; // hours is 00 to 11 in string format if (daynight == "am") { dateComponents.hour = Int(hours); } else { dateComponents.hour = Int(hours)! + 12; } // mins is 00 to 59 in string format dateCom...
Read more