How to load Kibana Visualizations with python script using REST requests

I want to automate creating Visualization and Dashboard on Kibana with a python script. I was able to do this with cURL command on terminal, PUTting data to 'http://localhost:9200/.kibana/visualization/visualization_name'.


However, I faced a problem when I added this snippet to my python script.

First I tried hard coding my cURL command to docstrings and passing it to terminal:

import subprocess
command = '''curl -XPUT http://localhost:9200/.kibana/visualization/test_visual -d'{"title":"test_visual","visState":"{\"aggs\":[{\"id\":\"1\",\"params\":{},\"schema\":\"metric\",\"type\":\"count\"},{\"id\":\"2\",\"params\":{\"field\":\"SYSTEM\",\"order\":\"desc\",\"orderBy\":\"1\",\"size\":100},\"schema\":\"segment\",\"type\":\"terms\"}],\"listeners\":{},\"params\":{\"addLegend\":true,\"addTooltip\":true,\"defaultYExtents\":false,\"shareYAxis\":true},\"type\":\"line\"}","description":"","version":1,"kibanaSavedObjectMeta":{"searchSourceJSON":"{\"index\":\"id_1\",\"query\":{\"query_string\":{\"analyze_wildcard\":true,\"query\":\"*\"}},\"filter\":[]}"}}' '''
output = subprocess.check_output(command, shell=True)
print(output)

This returned:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   731  100   313  100   418   4012   5358 --:--:-- --:--:-- --:--:--  6741
b'{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"failed to parse"}],"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":{"type":"not_x_content_exception","reason":"Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes"}},"status":400}'

I have also tried using requests library:

import requests
url = 'http://localhost:9200/.kibana/visualization/test_visual'
data = '''{"title":"test_visual","visState":"{\"aggs\":[{\"id\":\"1\",\"params\":{},\"schema\":\"metric\",\"type\":\"count\"},{\"id\":\"2\",\"params\":{\"field\":\"SYSTEM\",\"order\":\"desc\",\"orderBy\":\"1\",\"size\":100},\"schema\":\"segment\",\"type\":\"terms\"}],\"listeners\":{},\"params\":{\"addLegend\":true,\"addTooltip\":true,\"defaultYExtents\":false,\"shareYAxis\":true},\"type\":\"line\"}","description":"","version":1,"kibanaSavedObjectMeta":{"searchSourceJSON":"{\"index\":\"id_1\",\"query\":{\"query_string\":{\"analyze_wildcard\":true,\"query\":\"*\"}},\"filter\":[]}"}}'''
response = requests.put(url, data=data)
print(response)

Which also returned 400


I know that the cURL is correct as I was able to run this fine in shell script/terminal and was able to confirm from Kibana dashboard. Please help me understand my error. Thank you. I'm using ES 2.2.0 and Kibana 4.4.0

EDIT I realized that I can't run it in shell it either if I save the command in a variable ...

Answers


I figured this out yesterday, sooo I'll answer my own question because I know someone else will struggle with this at some point in time. And I need some more explanation regarding inconsistent behaviours in shell if someone with more knowledge stumble across this.


Working code:

import requests
url = 'http://localhost:9200/.kibana/visualization/test_visual'
data = """{"title":"test_visual","visState":"{\\"aggs\\":[{\\"id\\":\\"1\\",\\"params\\":{},\\"schema\\":\\"metric\\",\\"type\\":\\"count\\"},{\\"id\\":\\"2\\",\\"params\\":{\\"field\\":\\"SYSTEM\\",\\"order\\":\\"desc\\",\\"orderBy\\":\\"1\\",\\"size\\":100},\\"schema\\":\\"segment\\",\\"type\\":\\"terms\\"}],\\"listeners\\":{},\\"params\\":{\\"addLegend\\":true,\\"addTooltip\\":true,\\"defaultYExtents\\":false,\\"shareYAxis\\":true},\\"type\\":\\"line\\"}","description":"","version":1,"kibanaSavedObjectMeta":{"searchSourceJSON":"{\\"index\\":\\"id_1\\",\\"query\\":{\\"query_string\\":{\\"analyze_wildcard\\":true,\\"query\\":\\"*\\"}},\\"filter\\":[]}"}}"""
response = requests.put(url, data=data)
print(response)

So the changes I have made it to work were to escape the forward slash as well as the double quotes in visState value object and searchSourceJSON value object.


This string will still NOT WORK with

os.popen(command).read() 

nor

subprocess.check_ouput(command, shell=True)

Another important note I realized was that the behaviour will change depending on the shell you are using. I have done all my testing on MINGW64 (Git bash for Windows).

In MINGW64, you can copy paste this command directly into the shell:

curl -XPUT http://localhost:9200/.kibana/visualization/test_visual-d '{"title":"test_visual","visState":"{\"aggs\":[{\"id\":\"1\",\"params\":{},\"schema\":\"metric\",\"type\":\"count\"},{\"id\":\"2\",\"params\":{\"field\":\"SUBMITDATE\",\"order\":\"desc\",\"orderBy\":\"1\",\"size\":100},\"schema\":\"segment\",\"type\":\"terms\"}],\"listeners\":{},\"params\":{\"addLegend\":true,\"addTooltip\":true,\"defaultYExtents\":false,\"mode\":\"stacked\",\"shareYAxis\":true},\"type\":\"histogram\"}","description":"","version":1,"kibanaSavedObjectMeta":{"searchSourceJSON":"{\"index\":\"id_1\",\"query\":{\"query_string\":{\"analyze_wildcard\":true,\"query\":\"*\"}},\"filter\":[]}"}}'

However in Windows PowerShell (with cURL installed), you must triple the quotes for all strings inside data:

curl -XPUT http://localhost:9200/.kibana/visualization/test_visual -d '{"""title""":"""test_visual""","""visState""":"""{\\"""aggs\\""":[{\\"""id\\""":\\"""1\\""",\\"""params\\""":{},\\"""schema\\""":\\"""metric\\""",\\"""type\\""":\\"""count\\"""},{\\"""id\\""":\\"""2\\""",\\"""params\\""":{\\"""field\\""":\\"""SYSTEM\\""",\\"""order\\""":\\"""desc\\""",\\"""orderBy\\""":\\"""1\\""",\\"""size\\""":100},\\"""schema\\""":\\"""segment\\""",\\"""type\\""":\\"""terms\\"""}],\\"""listeners\\""":{},\\"""params\\""":{\\"""addLegend\\""":true,\\"""addTooltip\\""":true,\\"""defaultYExtents\\""":false,\\"""shareYAxis\\""":true},\\"""type\\""":\\"""line\\"""}""","""description""":"""""","""version""":1,"""kibanaSavedObjectMeta""":{"""searchSourceJSON""":"""{\\"""index\\""":\\"""id_1\\""",\\"""query\\""":{\\"""query_string\\""":{\\"""analyze_wildcard\\""":true,\\"""query\\""":\\"""*\\"""}},\\"""filter\\""":[]}"""}}'

That would mean you would have to modify the above solution to output the targeted string depending on the shell you are using.


In conclusion, if you are using Python, stick to Python libraries that can handle request. Try to use shells that are closest to Unix shells.


Need Your Help

Flex application font style is different between Flash Builder and MXMLC

apache-flex flash mxmlc

When I compile my project with MXMLC I get a light grey font; if I specify a CSS style Application { color: black;} it fixes some of the colors but not most of them. When I compile with Flash Build...

Optimize a rankings page using PHP and MySQL

php sql mysql database ranking

I could really use some help optimizing a table on my website that is used to display rankings. I have been reading a lot on how to optimize queries and how to properly use indexes but even after