#httpie

Some days I interact with REST APIs more than others. Ansible, python and curl, but what is most helpful is adhoc queries with HTTPie and using the Node.js json command.

NimbleOS has a public REST API for customers and partners to interact with. It has a token based authentication scheme where the token expire after a while.

I have arrays all over the place and many times in need of simply switch from one array to another to re-run the same statement or cross-reference a certain behavior between arrays and/or manipulate storage resources.

This blog post demonstrates how this can be accomplished in a simply manner. If you want to play along, install http and json as instructed.

First I have a "oneliner" shell script that needs to be sourced:

$ wget -q https://gist.githubusercontent.com/drajen/8be92279bdcb83a51d5c89bb3c0c90d0/raw/34d75b0a9dcf312716176bb5002ff80c8093ffd4/array_token.sh $ source array_token.sh Usage: source array_token.sh <array hostname or IP address> $ source array_token.sh sjc-array869.lab.nimblestorage.com Username: admin Password: http $API_URL/arrays $HEADER

The script spits out a shortened HTTPie command with the variables populated. It can simply be copied and pasted to interact with the array.

$ http $API_URL/arrays $HEADER HTTP/1.1 200 OK Connection: Keep-Alive Content-Type: application/json;charset=utf-8 Date: Fri, 18 Jan 2019 02:38:56 GMT Transfer-Encoding: chunked { "data": [ { "id": "0908e61269accae6a9000000000000000000000001", "name": "sjc-array869" } ], "endRow": 1, "startRow": 0, "totalRows": 1 }

Pretty neat. Now with the power of the Node.js JSON command, it’s quite trivial to get a single value from an object:

$ http $API_URL/arrays/0908e61269accae6a9000000000000000000000001 $HEADER | json data.version 5.0.5.0-583273-opt

A more elaborate example on how to find all offline volumes:

$ http $API_URL/volumes/detail $HEADER | json data | json -a name -c 'this.online == false' general-62959f81-de15-11e8-b8f9-525400910875.docker general-5b9bfbfb-dd22-11e8-b8f9-525400910875.docker general-43678a38-f988-11e8-b8f9-525400910875.docker general-9f9c0f8b-f989-11e8-b8f9-525400910875.docker general-69b5f072-f988-11e8-b8f9-525400910875.docker general-919c61ad-ddf7-11e8-b8f9-525400910875.docker general-cf283742-f987-11e8-b8f9-525400910875.docker general-04cd89a2-de15-11e8-b8f9-525400910875.docker general-d91fe30e-f97c-11e8-b8f9-525400910875.docker

HTTPie is also capable of using other HTTP verbs, such as PUT, POST, DELETE etc. In a completely overloaded example, I’ll fetch a volume ID, parse with json and xargs into a PUT operation that limits the IOPS of the volume.

$ http "$API_URL/volumes/detail?name=myvol1&fields=id" $HEADER \ | json data.0.id \ | xargs -n1 -I% http --verify=no --ignore-stdin PUT $API_URL/volumes/% $HEADER data:='{"limit_iops": 25000}' \ | json data.limit_iops 25000

Note: See the NimbleOS REST API Reference for valid fields that is PUT'able.

I came across this issue in httpie and started my investigation.

The problem is while pretty printing the json, output is alpha sorted because keys are hashed and user wanted to preserve the order. Then I made 3 comments to the issue. First comment was half correct and explained why it isn't possible to get the desired output, quickly I figured my assumptions were wrong and second comment explained what is actually happening, finally I proposed the solution. Since I made wrong assumptions and to make further debugging easy, I want to find easiest way to trace all functions/methods invocations.

I remember Noufal sharing small snippet in twitter, but I wanted to use trace and wrote small snippet(final one).

#! -*- coding: utf-8 -*- from httpie.core import main #import pdb import sys #import os #import trace #pdb.set_trace() main(args=sys.argv[1:])

First I ran the command python -m trace --trace httpie_test.py --pretty=all http://httpbin.org/get and it printed some 20 million lines. Then I got clever idea of using pdb.

I debugged with pdb for half an hour pressing s key and fed up. The pdb was beautiful like her, I was enjoying each line it was printing, it was like watching her speak and I was mesemerized. After half an hour I gave up and went back to trace command. Finally I figured out I can use --ignore-module from command line.

After one hour of spending time, final command looked like(scroll completely)

➜ snippets python -m trace --trace --ignore- module=os,sre_compile,sre_parse,zipfile,text_file,sysconfig,pkg_resources,re,posixpath,genericpath,decoder,hex_codec,socket,httplib,pkgutil,stat,token,style,calendar,spawn,util,collections,abc,decimal,lexer,StringIO,plist,argparse,_abcoll,structures,Queue,threading,_collections,urlparse,cookielib,platform,cookielib,terminal256,formatter,__init__,stringprep,_weakrefset,filter,encoder,codecs,latin,connectionpool,plugin,html,pygmentplugin,htmlentitydefs,__future__,weakref,UserDict,atexit,functools,base64,struct,hashlib,ssl,textwrap,six,exceptions,warnings,mimetools,tempfile,random,tempfile,rfc822,urllib,filepost,uuid,_endian,dyld,dylib,framework,io,poolmanager,pyopenssl,utils,cgi,netrc,shlex,compat,copy,numbers,locale,locale,unicode_escape,urllib2,Cookie,_LWPCookieJar,_MozillaCookieJar,ordered_dict,cookies,certs,ascii,status_codes,gettext,scanner,config,minicompat,domreg,minidom,xmlbuilder,NodeFilter,shutil,copy_reg,string,plistlib,_mapping,bbcode,img,Image,FixTk,ImageMode,ImagePalette,ImageColor,ImageDraw,ImageFont,latex,other,console,rtf,svg,terminal,solarized,getpass,pprint,downloads,idna,agile,web,unistring,functional,jvm,compiled,mimetypes httpie_test.py --pretty=all --verbose http://headers.jsontest.com/ >> op.txt ➜ snippets wc -l op.txt 2344 op.txt