Multi tool use

Kylin 2.3.1 Cluster - some nodes fail to query against cube

Problem: When I perform a simple SUM() query on my built cube, it runs sub-second on 1 cluster node, but the other 2 cluster nodes don't recognize a cube for that query and they run forever (so long that I cancel the query).

Context:
Version: Kylin 2.3.1
Mode: Clustered

I have created a Kylin Cube on top of a Fact table in Hive, and Built a data segment using a sample date range. My Kylin configuration is running as a 3 node cluster.

Node 1 is configured as a job & query server (in conf/kylin.properties the setting is: kylin.server.mode=all).

Nodes 2 and 3 are configured as query-only servers (in conf/kylin.properties the setting is: kylin.server.mode=query)

Once I have successfully built my cube with a data segment, I try to run a query like this in the Kylin UI Insight tab:

SELECT SUM(some_metric) AS value FROM my_fact_table


If I execute this query from the web UI on node 1 or node 3, the query goes into [executing] status forever.

If I execute the exact same query from the web UI on node 2, the query returns in 0.02 seconds.

So, my nodes 1 and 3 are rendered useless as end-points for querying.

Questions

See picture of results on node 2 and 3:

I've compared the kylin/logs/kylin.log files for node 1 (failing) and node 2 (working). Both logs matched each other message for message up until the following spot where node 1 fails... See below:

2018-07-02 16:38:25,629 DEBUG [Query eaf48991-94fd-40cd-9834-1097e79c6840-74] enumerator.OLAPEnumerator:120 : return TupleIterator...

2018-07-02 16:38:46,337 INFO [Scheduler 256150323 FetcherRunner-69] threadpool.DefaultScheduler:268 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 588 already succeed, 43 error, 49 discarded, 0 others

2018-07-02 16:39:05,911 INFO [kylin-coproc--pool3-t1] client.RpcRetryingCaller:146 : Call exception, tries=10, retries=35, started=68253 ms ago, cancelled=false, msg=java.io.IOException: Message missing required fields: compressedRows, stats
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2195)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:187)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:167)
Caused by: com.google.protobuf.UninitializedMessageException: Message missing required fields: compressedRows, stats
at com.google.protobuf.AbstractMessage$Builder.newUninitializedMessageException(AbstractMessage.java:770)
at org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitResponse$Builder.build(CubeVisitProtos.java:5019)
at org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitResponse$Builder.build(CubeVisitProtos.java:4949)
at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:7866)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:1980)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:1962)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32389)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2150)


So, the client.RpcRetryingCaller (I assume that's the Kylin server making an RPC call to HBase) is failing. The error message is:

java.io.IOException: Message missing required fields: compressedRows, stats


By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.