How could we run the Kafka example in the official Spark project?


How could we run the Kafka example in the official Spark project?



I try to run the simple Kafka example in Spark project.



The spark built properly, so I can run most of the examples, except this Kafka example.



When I run the following command:


bin/run-example streaming.JavaDirectKafkaWordCount localhost:9092 test



I got the following error:


Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/streaming/kafka010/LocationStrategies



Any idea how could we run the Kafka Example properly?



(Kafka client running on port 9092. Using the latest Spark and Kafka, and Java 8)


9092





same problem, if you find a solution...
– Jokin
Jun 1 at 12:41





ibm.com/support/knowledgecenter/en/SSWTQQ_2.0.0/solnguide/…
– mangusta
Jun 1 at 15:57





Have you found kafka010 jar on the Spark library folder?
– cricket_007
Jun 2 at 0:15


kafka010




1 Answer
1



In my project, finally I connected together Kafka and Spark properly. I realised, using the local mode in Spark script is quite useful during development. Much easier to debug and practice.


local



You can find a detailed, dockerized implementation here: https://github.com/zoltan-nz/kafka-spark-project



Related notes: https://github.com/zoltan-nz/kafka-spark-project/blob/master/SparkStreamer/README.md#experiment-06---using-local






By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

-

Popular posts from this blog

api-platform.com Unable to generate an IRI for the item of type

api-platform.com Unable to generate an IRI for the item of type I'm trying to setup my first api-platform instance and have run into some presumably noob questions around IDE. What i'm trying to do is setting up an register and reset entity in order to deal with registrations and password reminders from an app. And then have the entity's passed to a custom controller to do some checks, balances and mails afterwards (haven’t gotten around to that yet). But I keep running into this error when posting to my custom entities: { "@context": "/api/contexts/Error", "@type": "hydra:Error", "hydra:title": "An error occurred", "hydra:description": "Unable to generate an IRI for the item of type "AppXXXUserBundleEntityRegister"", "trace": [ { "namespace": "", "short_class": "", "c
Read more

How to set up datasource with Spring for HikariCP?

How to set up datasource with Spring for HikariCP? Hi I'm trying to use HikariCP with Spring for connection pool. I'm using jdbcTempLate and JdbcdaoSupport. This is my spring configuration file for datasource: <bean id="dataSource" class="com.zaxxer.hikari.HikariDataSource"> <property name="dataSourceClassName" value="oracle.jdbc.driver.OracleDriver"/> <property name="dataSource.url" value="jdbc:oracle:thin:@localhost:1521:XE"/> <property name="dataSource.user" value="username"/> <property name="dataSource.password" value="password"/> </bean> But unfortunately the following error message is generating: Cannot resolve reference to bean 'dataSource' while setting bean property 'dataSource'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dataSource'
Read more

PHP contact form sending but not receiving emails

PHP contact form sending but not receiving emails I have created a contact form using PHP to send the data. The form seems to send fine as there are no errors and success message appears, however I am not receving the emails into the designated inbox. I am using PHPMailer as I originally tried just using the php 'mail' command which I now understand is a bit hit and miss. I cannot see why I would not be receving the emails so I would be very grateful for any help that could be given. I'm quite new to PHP so please be patient with me :) <?php use PHPMailerPHPMailerPHPMailer; use PHPMailerPHPMailerException; $msg = ""; if (isset($_POST['submit'])) { require 'phpmailer/src/Exception.php'; require 'phpmailer/src/PHPMailer.php'; require 'phpmailer/src/SMTP.php'; function sendemail ($to, $from, $body) { $mail = new PHPMailer(true); $mail->setFrom($from); $mail->addAddress($to); $ma
Read more