filebeat -> logstash -> elasticsearch(AWS), filebeat timeout connection error

Errors

2017-12-19T16:27:12+09:00 ERR  Failed to publish events caused by: read tcp [::1]:48818->[::1]:5044: i/o timeout

2017-12-19T16:27:12+09:00 ERR  Failed to publish events caused by: client is not connected

2017-12-19T16:27:13+09:00 ERR  Failed to publish events: client is not connected

In my case,

It is caused from elasticsearch error.

This is logstash log. (see logstash.yml. "log level", and change it to info)

[2017-12-19T16:53:22,168][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"index_create_block_exception", "reason"=>"blocked by: [FORBIDDEN/10/cluster create-index blocked (api)];"})

My AWS ES status was yellow(you should have minimum two ES instances. I had only one ES instance)

Just add one more ES or delete your ES domain and recreate it.