Tuesday, April 29, 2014

How To Analyze HAProxy Logs

Sometimes it is very difficult to analyse the HaProxy Logs manually . Very few people know about the small tool name halog , it gets shipped with HaProxy itself.
HALog is a small and very powerful tool to analyze HaProxy log lines.Installation is pretty simple, as described bellow:
  • cd /usr/src
  • wget http://haproxy.1wt.eu/download/1.5/src/devel/haproxy-1.5-dev11.tar.gz
  • tar xzf haproxy-1.5-dev11.tar.gz
  • cd haproxy-1.5-dev11/contrib/halog
  • make
  • cp halog /usr/bin/
and we are done and ready to analyse the HaProxy Logs. Below are few examples 
  1. List servers by number of requests treated
The command below lists the servers by the number of requests they treated. The server name is
prefixed by the backend name.
 
The eighth columns "tot_req" gives the number of requests treated by the server.
 
cat filename.log | halog -srv -H -q |awk ’NR==1; NR > 1 {print $0 | "sort -n -r -k 9"}’ |column -t
 
       2. List servers by response time
 
The command below lists the servers by response time. The server name is prefixed by the backend
name.
The response time is in milliseconds and the latest columns "avg_rt" gives the average response time
for all the URLs forwarded to this server in this backend.
 
cat filename.log | halog -srv -H -q |awk ’NR==1; NR > 1 {print $0 | "sort -n -r -k 12"}’ |column -t
 
      3. List servers by application errors: HTTP status code 5xx
 
The command below lists the servers by number of application errors. The server name is prefixed by
the backend name.
The sixth column "5xx" gives the number of application errors generated by the server.
 
cat filename.log |halog -srv -H -q | awk ’NR==1; NR > 1 {print $0 | "sort -n -r -k 6"}’ | column -t
 
      4. List servers by errors
 
The command below lists the servers by number of errors not related to the application. The server
name is prefixed by the backend name.
 
cat filename.log |halog -srv -H -q | awk ’NR==1; NR > 1 {print $0 | "sort -n -r -k 5"}’ | column -t
 
    5. List URLs by server computation time
The command below lists the URLs by the average computation time, whatever the server which treated
it.
The sixth column "okavg" provides the URL average computation time in milliseconds.
 
cat filename.log | halog -ut -H -q | column -t
 
    6. List URLs by errors
 
The command below lists the URLs by the number of errors they have generated, whatever the server
which treated it or the type of error.
The second column "err" provides the number of errors generated by the given URL (latest column).
 
cat filename.log | halog -ue -H -q | column -t
 
   7 . List URLs by missing files: HTTP status code 404
The command below lists the URLs by the number of missing files error they have generated, whatever
the server which treated it.
The first column "req" provides the number of 404 returned for the given URL (latest column).
 
cat filename.log | halog -u -H -q -hs 404 | column -t
 
    8. List URLs by number of request
The command below lists the URLs by the number of time they have been requested on the platform.
The first column "req" provides the number of time the URLs was called.
 
cat filename.log | halog -u -H -q | awk ’NR==1; NR > 1 {print $0 | "sort -n -r -k 1"}’ | column -t

Friday, April 25, 2014

Fabric - Pythonic remote execution

Fabric is a Python (2.5-2.7) library and command-line tool for streamlining the use of SSH for application deployment or systems administration tasks.

It provides a basic suite of operations for executing local or remote shell commands (normally or via sudo) and uploading/downloading files, as well as auxiliary functionality such as prompting the running user for input, or aborting execution.

Typical use involves creating a Python module containing one or more functions, then executing them via the fab command-line tool. 

How to install fabric on Linux Boxes.

Fabric is best installed via pip (highly recommended) or easy_install (older, but still works fine), e.g.:
$ pip install fabric
You may also opt to use your operating system’s package manager; the package is typically called fabric or python-fabric. E.g.:
$ sudo apt-get install fabric
Advanced users wanting to install a development version may use pip to grab the latest master branch (as well as the dev version of the Paramiko dependency):
$ pip install paramiko==dev
$ pip install fabric==dev

Dependencies

In order for Fabric’s installation to succeed, you will need four primary pieces of software:
  • the Python programming language;
  • the setuptools packaging/installation library;
  • the Python Paramiko SSH library;
  • and Paramiko’s dependency, the PyCrypto cryptography library.
and, if using the parallel execution mode:
If you’re using Paramiko 1.12 or above, you will also need an additional dependency for Paramiko:
Below are my 2 fabfile 

https://github.com/abhishekamralkar/Python/blob/master/myfabfile.py

https://github.com/abhishekamralkar/Python/blob/master/myfabfile2.py

To run get_version on both roles (webservers and databases);

fab get_version