Elasticsearch connection refused while kibana is trying to connect

I am trying to run ELK stack using docker container. But I am getting error that kibana is unable to make connection with elasticsearch.

kibana_1 | {"type":"log","@timestamp":"2018-06-22T19:31:38Z","tags":["error","elasticsearch","admin"],"pid":12,"message":"Request error, retryingnHEAD http://elasticsearch:9200/ => connect ECONNREFUSED 172.18.0.2:9200"}
kibana_1 | {"type":"log","@timestamp":"2018-06-22T19:31:38Z","tags":["status","plugin:console@5.6.9","info"],"pid":12,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibana_1 | {"type":"log","@timestamp":"2018-06-22T19:31:38Z","tags":["warning","elasticsearch","admin"],"pid":12,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibana_1 | {"type":"log","@timestamp":"2018-06-22T19:31:38Z","tags":["warning","elasticsearch","admin"],"pid":12,"message":"No living connections"}
kibana_1 | {"type":"log","@timestamp":"2018-06-22T19:31:38Z","tags":["status","plugin:elasticsearch@5.6.9","error"],"pid":12,"state":"red","message":"Status changed from yellow to red - Unable to connect to Elasticsearch at http://elasticsearch:9200.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
kibana_1 | {"type":"log","@timestamp":"2018-06-22T19:31:38Z","tags":["status","plugin:metrics@5.6.9","info"],"pid":12,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
elasticsearch_1 | [2018-06-22T19:31:38,182][INFO ][o.e.d.DiscoveryModule ] [g8HPieb] using discovery type [zen]
kibana_1 | {"type":"log","@timestamp":"2018-06-22T19:31:38Z","tags":["status","plugin:timelion@5.6.9","info"],"pid":12,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibana_1 | {"type":"log","@timestamp":"2018-06-22T19:31:38Z","tags":["listening","info"],"pid":12,"message":"Server running at http://0.0.0.0:5601"}
kibana_1 | {"type":"log","@timestamp":"2018-06-22T19:31:38Z","tags":["status","ui settings","error"],"pid":12,"state":"red","message":"Status changed from uninitialized to red - Elasticsearch plugin is red","prevState":"uninitialized","prevMsg":"uninitialized"}
elasticsearch_1 | [2018-06-22T19:31:38,634][INFO ][o.e.n.Node ] initialized
elasticsearch_1 | [2018-06-22T19:31:38,634][INFO ][o.e.n.Node ] [g8HPieb] starting ...
elasticsearch_1 | [2018-06-22T19:31:38,767][INFO ][o.e.t.TransportService ] [g8HPieb] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}
elasticsearch_1 | [2018-06-22T19:31:38,776][WARN ][o.e.b.BootstrapChecks ] [g8HPieb] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
logstash_1 | log4j:WARN No appenders could be found for logger (io.netty.util.internal.logging.InternalLoggerFactory).
logstash_1 | log4j:WARN Please initialize the log4j system properly.
logstash_1 | log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
**logstash_1 | {:timestamp=>"2018-06-22T19:31:40.555000+0000", :message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :level=>:error}**
kibana_1 | {"type":"log","@timestamp":"2018-06-22T19:31:40Z","tags":["warning","elasticsearch","admin"],"pid":12,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibana_1 | {"type":"log","@timestamp":"2018-06-22T19:31:40Z","tags":["warning","elasticsearch","admin"],"pid":12,"message":"No living connections"}


Here is the content of my docker-comp

version: "2.0"

services:

logstash:
image: logstash:2
ports:
- "5044:5044"
volumes:
- ./:/config
command: logstash -f /config/logstash.conf
links:
- elasticsearch
depends_on:
- elasticsearch

elasticsearch:
image: elasticsearch:5.6.9
ports:
- "9200:9200"
volumes:
- "./es_data/es_data:/usr/share/elasticsearch/data/"

kibana:
image: kibana:5
ports:
- "5601:5601"
links:
- elasticsearch
environment:
ELASTICSEARCH_URL: http://elasticsearch:9200
depends_on:
- elasticsearch


Content of my logstash.conf

input { beats { port => 5044 } }

output {
elasticsearch {
hosts => ["localhost:9200"]
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
stdout {
codec => rubydebug
}
}


I have curl on elasticsearch container and kibana container and it looks good to me

{
"name" : "g8HPieb",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "XxH0TcAmQcGqprf6s7TJEQ",
"version" : {
"number" : "5.6.9",
"build_hash" : "877a590",
"build_date" : "2018-04-12T16:25:14.838Z",
"build_snapshot" : false,
"lucene_version" : "6.6.1"
},
"tagline" : "You Know, for Search"
}


curl localhost:9200/_cat/indices?pretty

yellow open .kibana GIBmXdlRQJmI67oq5r4oCg 1 1 1 0 3.2kb 3.2kb


After increasing virtual memory size

root@sfbp19:~/dockerizing-jenkins# sysctl -p
vm.max_map_count = 262144
root@sfbp19:~/dockerizing-jenkins# docker-compose -f docker-compose-elk.yml up

Creating network "dockerizingjenkins_default" with the default driver
Creating dockerizingjenkins_elasticsearch_1
Creating dockerizingjenkins_logstash_1
Creating dockerizingjenkins_kibana_1
Attaching to dockerizingjenkins_elasticsearch_1, dockerizingjenkins_kibana_1, dockerizingjenkins_logstash_1
elasticsearch_1 | [2018-06-26T19:08:19,294][INFO ][o.e.n.Node ] initializing ...
elasticsearch_1 | [2018-06-26T19:08:19,363][INFO ][o.e.e.NodeEnvironment ] [PVmTsqv] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/mapper/sfbp19--vg-root)]], net usable_space [671.9gb], net total_space [789.2gb], spins? [possibly], types [ext4]
elasticsearch_1 | [2018-06-26T19:08:19,364][INFO ][o.e.e.NodeEnvironment ] [PVmTsqv] heap size [1.9gb], compressed ordinary object pointers [true]
elasticsearch_1 | [2018-06-26T19:08:19,369][INFO ][o.e.n.Node ] node name [PVmTsqv] derived from node ID [PVmTsqv3QnyS3sQarPcJ-A]; set [node.name] to override
elasticsearch_1 | [2018-06-26T19:08:19,369][INFO ][o.e.n.Node ] version[5.6.9], pid[1], build[877a590/2018-04-12T16:25:14.838Z], OS[Linux/4.4.0-31-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_171/25.171-b11]
elasticsearch_1 | [2018-06-26T19:08:19,369][INFO ][o.e.n.Node ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
elasticsearch_1 | [2018-06-26T19:08:20,040][INFO ][o.e.p.PluginsService ] [PVmTsqv] loaded module [aggs-matrix-stats]
elasticsearch_1 | [2018-06-26T19:08:20,040][INFO ][o.e.p.PluginsService ] [PVmTsqv] loaded module [ingest-common]
elasticsearch_1 | [2018-06-26T19:08:20,040][INFO ][o.e.p.PluginsService ] [PVmTsqv] loaded module [lang-expression]
elasticsearch_1 | [2018-06-26T19:08:20,040][INFO ][o.e.p.PluginsService ] [PVmTsqv] loaded module [lang-groovy]
elasticsearch_1 | [2018-06-26T19:08:20,040][INFO ][o.e.p.PluginsService ] [PVmTsqv] loaded module [lang-mustache]
elasticsearch_1 | [2018-06-26T19:08:20,040][INFO ][o.e.p.PluginsService ] [PVmTsqv] loaded module [lang-painless]
elasticsearch_1 | [2018-06-26T19:08:20,040][INFO ][o.e.p.PluginsService ] [PVmTsqv] loaded module [parent-join]
elasticsearch_1 | [2018-06-26T19:08:20,040][INFO ][o.e.p.PluginsService ] [PVmTsqv] loaded module [percolator]
elasticsearch_1 | [2018-06-26T19:08:20,041][INFO ][o.e.p.PluginsService ] [PVmTsqv] loaded module [reindex]
elasticsearch_1 | [2018-06-26T19:08:20,041][INFO ][o.e.p.PluginsService ] [PVmTsqv] loaded module [transport-netty3]
elasticsearch_1 | [2018-06-26T19:08:20,041][INFO ][o.e.p.PluginsService ] [PVmTsqv] loaded module [transport-netty4]
elasticsearch_1 | [2018-06-26T19:08:20,041][INFO ][o.e.p.PluginsService ] [PVmTsqv] no plugins loaded
kibana_1 | {"type":"log","@timestamp":"2018-06-26T19:08:20Z","tags":["status","plugin:kibana@5.6.9","info"],"pid":13,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibana_1 | {"type":"log","@timestamp":"2018-06-26T19:08:20Z","tags":["status","plugin:elasticsearch@5.6.9","info"],"pid":13,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
kibana_1 | {"type":"log","@timestamp":"2018-06-26T19:08:20Z","tags":["error","elasticsearch","admin"],"pid":13,"message":"Request error, retryingnHEAD http://elasticsearch:9200/ => connect ECONNREFUSED 172.18.0.2:9200"}
kibana_1 | {"type":"log","@timestamp":"2018-06-26T19:08:20Z","tags":["warning","elasticsearch","admin"],"pid":13,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibana_1 | {"type":"log","@timestamp":"2018-06-26T19:08:20Z","tags":["warning","elasticsearch","admin"],"pid":13,"message":"No living connections"}
kibana_1 | {"type":"log","@timestamp":"2018-06-26T19:08:20Z","tags":["status","plugin:console@5.6.9","info"],"pid":13,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibana_1 | {"type":"log","@timestamp":"2018-06-26T19:08:20Z","tags":["status","plugin:elasticsearch@5.6.9","error"],"pid":13,"state":"red","message":"Status changed from yellow to red - Unable to connect to Elasticsearch at http://elasticsearch:9200.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
kibana_1 | {"type":"log","@timestamp":"2018-06-26T19:08:20Z","tags":["status","plugin:metrics@5.6.9","info"],"pid":13,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibana_1 | {"type":"log","@timestamp":"2018-06-26T19:08:21Z","tags":["status","plugin:timelion@5.6.9","info"],"pid":13,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibana_1 | {"type":"log","@timestamp":"2018-06-26T19:08:21Z","tags":["listening","info"],"pid":13,"message":"Server running at http://0.0.0.0:5601"}
kibana_1 | {"type":"log","@timestamp":"2018-06-26T19:08:21Z","tags":["status","ui settings","error"],"pid":13,"state":"red","message":"Status changed from uninitialized to red - Elasticsearch plugin is red","prevState":"uninitialized","prevMsg":"uninitialized"}
elasticsearch_1 | [2018-06-26T19:08:21,190][INFO ][o.e.d.DiscoveryModule ] [PVmTsqv] using discovery type [zen]
elasticsearch_1 | [2018-06-26T19:08:21,654][INFO ][o.e.n.Node ] initialized
elasticsearch_1 | [2018-06-26T19:08:21,654][INFO ][o.e.n.Node ] [PVmTsqv] starting ...
elasticsearch_1 | [2018-06-26T19:08:21,780][INFO ][o.e.t.TransportService ] [PVmTsqv] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}
logstash_1 | log4j:WARN No appenders could be found for logger (io.netty.util.internal.logging.InternalLoggerFactory).
logstash_1 | log4j:WARN Please initialize the log4j system properly.
logstash_1 | log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
kibana_1 | {"type":"log","@timestamp":"2018-06-26T19:08:23Z","tags":["warning","elasticsearch","admin"],"pid":13,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibana_1 | {"type":"log","@timestamp":"2018-06-26T19:08:23Z","tags":["warning","elasticsearch","admin"],"pid":13,"message":"No living connections"}
logstash_1 | {:timestamp=>"2018-06-26T19:08:23.572000+0000", :message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :level=>:error}
logstash_1 | {:timestamp=>"2018-06-26T19:08:23.790000+0000", :message=>"Pipeline main started"}
elasticsearch_1 | [2018-06-26T19:08:24,837][INFO ][o.e.c.s.ClusterService ] [PVmTsqv] new_master {PVmTsqv}{PVmTsqv3QnyS3sQarPcJ-A}{coD5A4HyR7-1MedSq8dFUQ}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)[, ]
elasticsearch_1 | [2018-06-26T19:08:24,869][INFO ][o.e.h.n.Netty4HttpServerTransport] [PVmTsqv] publish_address {172.18.0.2:9200}, bound_addresses {0.0.0.0:9200}
elasticsearch_1 | [2018-06-26T19:08:24,870][INFO ][o.e.n.Node ] [PVmTsqv] started
elasticsearch_1 | [2018-06-26T19:08:24,989][INFO ][o.e.g.GatewayService ] [PVmTsqv] recovered [1] indices into cluster_state
elasticsearch_1 | [2018-06-26T19:08:25,148][INFO ][o.e.c.r.a.AllocationService] [PVmTsqv] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.kibana][0]] ...]).
kibana_1 | {"type":"log","@timestamp":"2018-06-26T19:08:26Z","tags":["status","plugin:elasticsearch@5.6.9","info"],"pid":13,"state":"green","message":"Status changed from red to green - Kibana index ready","prevState":"red","prevMsg":"Unable to connect to Elasticsearch at http://elasticsearch:9200."}
kibana_1 | {"type":"log","@timestamp":"2018-06-26T19:08:26Z","tags":["status","ui settings","info"],"pid":13,"state":"green","message":"Status changed from red to green - Ready","prevState":"red","prevMsg":"Elasticsearch plugin is red"}


========================filebeat.yml==============================

filebeat.inputs:
- type: log
enabled: true
paths:
- /jenkins/gerrit_volume/logs/*_log
#============================= Filebeat modules ===============================
filebeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/*.yml

# Set to true to enable config reloading
reload.enabled: false
#==================== Elasticsearch template setting ==========================
setup.template.settings:
index.number_of_shards: 3
#============================== Kibana =====================================
setup.kibana:
#host: "localhost:5601"
#----------------------------- Logstash output --------------------------------
output.logstash:
# The Logstash hosts
hosts: ["10.1.9.69:5044"]

logging.level: debug


1 Answer
1

It looks like this is an elasticsearch problem from your logging, preventing ES from initializing. This line:

elasticsearch_1 | [2018-06-22T19:31:38,776][WARN ][o.e.b.BootstrapChecks ] [g8HPieb] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]


You can bump it up temporarily with the following command:

sysctl -w vm.max_map_count=262144


Or set it permanently by adding the following line to /etc/sysctl.conf and running sysctl -p to pick up the config if you're on a live instance:

vm.max_map_count=262144


Since you're doing this in a Docker container you probably want to just spin up with the latter option in /etc/sysctl.conf.

Reference: https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.