HTTPS From Java to Logstash

The goal: securely sending data from a server that has access to a rabbitMq based system to a remote server (and then to s3)

How would you do that?

Due to a special protocol when accessing the rabbitMq system, we decided to do that with a custom consuming java code in the Rabbit side and with Logstash listening on 443 for HTTPS posts.

We really wanted to do that with Logstash to Logstash but couldn't due to special requirements.

Why https? We tried TCP over SSL but when a new connection is being established, and then the listener Logstash is going down, we had to work harder in order to re-establish the connection from the java side. And in addition, no response code is being sent back when working over tcp with no http.

We though of using Logstash Lumberjack on the listener side, but then you have to use Logstash with Lumberjack output.

Step #1: Creating certificates (if you don't have any) and import them into the keystore.

On the Logstash box:

1.  keytool -genkey -keyalg RSA -alias mycert -keystore keystore.jks -storepass 123pass -ext SAN=ip:172.18.22.22,ip:172.22.22.24  -validity 360 -keysize 2048 
2. The 2 ips are the ips that we want the java box to trust when it is connecting Logstash. In that case we had to pass via another routing box in the middle.
3. Export the created certificate for the java consuming client use.
4. keytool -exportcert -file client.er -alias mysert -keystore keystore.jks 

On the Java side:

1. Create a truststore with the exported certificate:
1. keytool -importcert -file client.cer  -alias mycert -keystore truststore.jks
2. You can  import another certificates if needed

Step #2: Java app

I will focus on the https part with basic authentication.

In order to send data through https you can use the next code that is based on apache http client (it was harder than I expected it would be to find the right way to do that with SSL):

Imports:

 import org.apache.commons.codec.binary.Base64;  
 import org.apache.http.client.ClientProtocolException;  
 import org.apache.http.client.config.RequestConfig;  
 import org.apache.http.client.methods.CloseableHttpResponse;  
 import org.apache.http.client.methods.HttpPost;  
 import org.apache.http.conn.HttpHostConnectException;  
 import org.apache.http.conn.ssl.SSLConnectionSocketFactory;  
 import org.apache.http.conn.ssl.TrustSelfSignedStrategy;  
 import org.apache.http.entity.ByteArrayEntity;  
 import org.apache.http.impl.client.CloseableHttpClient;  
 import org.apache.http.impl.client.HttpClients;  
 import org.apache.http.ssl.SSLContexts;  

Some setups before sending (We setup the apache https client. It also has a basic authentication headers with username and password).

We point the java keystore location that we have created on the java side, in part #1.

   httpclient = HttpClients.createDefault();  

     SSLContext sslcontext = SSLContexts.custom().loadTrustMaterial(new File(truststore_location), truststore_password.toCharArray(),  
         new TrustSelfSignedStrategy())  
         .build();  
     // Allow TLSv1 protocol only  
     SSLConnectionSocketFactory sslsf = new SSLConnectionSocketFactory(sslcontext,null, null, SSLConnectionSocketFactory.getDefaultHostnameVerifier());  
     httpclient = HttpClients.custom().setSSLSocketFactory(sslsf).build();  
     final RequestConfig params = RequestConfig.custom().setConnectTimeout(5000).setSocketTimeout(5000).build();  
     httppost = new HttpPost("https://"+logstashUrl);  
     httppost.setConfig(params);  
     String credentials = this.httpuser+":"+httppassword;  
     byte[] encoding=Base64.encodeBase64(credentials.getBytes());  
     String authStringEnc = new String(encoding);  
     httppost.setHeader("Authorization", "Basic " + authStringEnc);  

Sending chunks of data:

  ByteArrayEntity postDataEntity = new ByteArrayEntity(data.getBytes());  
 httppost.setEntity(postDataEntity);  
 //Actual post to remote host  
 CloseableHttpResponse httpResponse = httpclient.execute(httppost);  
 BufferedReader reader = new BufferedReader(new InputStreamReader(httpResponse.getEntity().getContent()));  
 String inputLine;  
 StringBuffer stringResponse = new StringBuffer();  
 while ((inputLine = reader.readLine()) != null) {  
   stringResponse.append(inputLine);  
  }  
 reader.close();  

Part #3: Logstash with HTTPS INPUT
h
That's the easy and well documented part.
A sample of configuration for that kind of logstash:

 input {  
  http {  
   host => "127.0.0.1" # default: 0.0.0.0  
   port => 443   
   keystore=>"/home/user/keystore.jks"  
   keystore_password => "123pass"  
   ssl => true  
   password => "my_basic_auth_pass"  
   user => "my_basic_auth_user"  
  }   
 }   

Path #5: Logstash output (Epilogstash)

So we needed the data to get to S3 and amazon Kinesis.
Logstash, as the component that gets the data via https gives us lots of flexibility.
It supports S3 out of the box. The problem is that it doesn't support S3 server side encryption. So it wasn't good for us.
There is a github project for Kinesis output. We preferred not using that.

The temporary solution is to wrtie to files with logstash,
and then reading the files with python script and sending to s3 using the aws cli.
Kinesis has a nice agent that is able to consume a directory and to send new lines to kinesis.

That's it. Good luck!