Monday, September 28, 2015

ngxtop for analyze Nginx log

https://github.com/lebinh/ngxtop
ngxtop parses your nginx access log and outputs useful, top-like, metrics of your nginx server. So you can tell what is happening with your server in real-time.
ngxtopは、nginxのログをリアルタイムに解析して、topコマンドのように表示してくれるプログラムです。
オプションをつけたりして工夫すると、Apacheのlogも扱えることができるようです。
sshログインして、どこにアクセスが来ているのかな?というふうに、ぱぱっと見ることができるので、使えるようにしておいて損はしないと思います。

以下、インストールメモです。pipで入るので簡単です。
環境: CentOS 6
# yum install python-setuptools
# easy_install pip
# pip install ngxtop
# ngxtop -h
通常のログフォーマットの他に、以下の様な、LTSVフォーマットのログも解析できます。
ngxtopがnginx.confをパースしてくれるので、とくにオプションの指定は必要ないです。
# cat /etc/nginx/nginx.conf
~~~
    log_format  ltsv  "time:$time_local"
                      "\thost:$remote_addr"
                      "\tforwardedfor:$http_x_forwarded_for"
                      "\treq:$request"
                      "\tstatus:$status"
                      "\tsize:$body_bytes_sent"
                      "\treferer:$http_referer"
                      "\tua:$http_user_agent"
                      "\treqtime:$request_time"
                      "\tapptime:$upstream_response_time"
                      "\tvhost:$host";

    access_log  /var/log/nginx/access.log ltsv;
~~~
ngxtop infoと打つと、どのログを解析するのか見ることができます。
# ngxtop info
nginx configuration file:
  /etc/nginx/nginx.conf
access log file:
  /var/log/nginx/access.log
access log format:
  time:$time_local\thost:$remote_addr\tforwardedfor:$http_x_forwarded_for\treq:$request\tstatus:$status\tsize:$body_bytes_sent\treferer:$http_referer\tua:$http_user_agent\treqtime:$request_time\tapptime:$upstream_response_time\tvhost:$host
available variables:
  body_bytes_sent, host, http_referer, http_user_agent, http_x_forwarded_for, remote_addr, request, request_time, status, time_local, upstream_response_time
違うパスのログファイルを解析するときは、オプションを付ければOKです。
Options:
    -l , --access-log   access log file to parse.

Title pic from Linux Screenshots Log Monitoring Tool - ngxtop

Install and update aws cli

AWS Command Line Interface Documentation
http://aws.amazon.com/documentation/cli/

This is how to for install new version of AWS CLI.
Environment: CentOS6

Install aws cli with Python pip
# yum install python-setuptools
# easy_install pip
# pip install awscli
# aws --version
# aws-cli/1.8.7 Python/2.6.6 Linux/2.6.32
# aws configure
AWS Access Key ID [None]: xxxx
AWS Secret Access Key [None]: xxxx
Default region name [None]: ap-northeast-1
Default output format [None]: 
Check local config file
# cat .aws/config 
[default]
region = ap-northeast-1
# cat .aws/credentials 
[default]
aws_access_key_id = xxx
aws_secret_access_key = xxx
Do test with s3 ls
# aws s3 ls 
2015-09-11 08:31:13 my-cloudtrail
2015-04-08 03:38:40 my-public
2015-04-14 14:47:05 my-spark-test

Update aws cli by pip
# aws --version
aws-cli/1.7.25 Python/2.6.6 Linux/2.6.32
# pip install --upgrade awscli
# aws --version
aws-cli/1.8.7 Python/2.6.6 Linux/2.6.32
# aws configure
AWS Access Key ID [None]: xxxx
AWS Secret Access Key [None]: xxxx
Default region name [None]: ap-northeast-1
Default output format [None]: 

Title image from Emanuele Cento Canesio

Tuesday, September 15, 2015

Nagios plugin for Amazon SQS

https://github.com/takeshiyako2/nagios-check_sqs_messages

This is nagios plugin for message queue of Amazon SQS.
You can check count of message queue.

Setup
# gem install aws-sdk
# gem list | grep aws-sdk
aws-sdk (2.1.20)
aws-sdk-core (2.1.20)
aws-sdk-resources (2.1.20)
If you use aws-sdk-v1, use check_sqs_messages_aws_sdk_v1.rb

Usage
ruby check_sqs_messages.rb -c 10 -w 5 -q  -a  -s  -r 
Example
ruby check_sqs_messages.rb -c 10 -w 5 -q my-queue-name -a XXXX -s YYYY -r ap-northeast-1
OK - my-queue-name message count is 3 |message=3

See also.
http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/sqs-metricscollected.html

Titile pic form Alexandre Duret-Lutz Queue

Thursday, September 10, 2015

IAMポリシーを利用して、Amazon S3にIPアドレス制限

Amazon S3の特定のバケツに、IAMポリシーを利用して、IPアドレス制限を掛ける方法です。
特定のユーザにAccess Key IDを発行して、S3の操作を制限させることができます。
たとえば、社外のユーザにS3にファイルをアップロードしてもらったりするようなシチュエーションがあったときなどに、使える方法だと思います。
以下、設定方法です。

IAMサービスを選択してポリシーを作っていきます。
Services -> IAM -> Policies -> Create Policy -> Create Your Own Policy

ポリシーの内容は、以下のようにします。

Policy Name:
好きな名前を入力します。
この例では、S3-IP-Restrictions とします。

Policy Document:
Statementを3つ書きます。
上から2つは、AWS CLIで操作できるようにするためのものです。もし、AWS CLIからアクセスせず、なんらかのプログラムを書いてアクセスするならば、一番下のStatementだけ書けばOKです。
"aws:SourceIp"項目では、IPアドレス100.100.100.101と100.100.100.102を許可します。設定では、後ろにサブネットマスク/32もつけてあげます。
"Resource"には、アクセス許可をするS3のバケツを指定します。この例では my-s3-bucket というバケツを使います。
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1441865835000",
            "Effect": "Allow",
            "Action": [
                "s3:ListAllMyBuckets" 
            ],
            "Condition": {
                "IpAddress": {
                    "aws:SourceIp": [
                        "100.100.100.101/32",
                        "100.100.100.102/32"
                    ]
                }
            },
            "Resource": [
                "arn:aws:s3:::*" 
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation" 
            ],
            "Condition": {
                "IpAddress": {
                    "aws:SourceIp": [
                        "100.100.100.101/32",
                        "100.100.100.102/32"
                    ]
                }
            },
            "Resource": "arn:aws:s3:::my-s3-bucket" 
        },
        {
            "Effect": "Allow",
            "Action": [
                "*" 
            ],
            "Condition": {
                "IpAddress": {
                    "aws:SourceIp": [
                        "100.100.100.101/32",
                        "100.100.100.102/32"
                    ]
                }
            },
            "Resource": [
                "arn:aws:s3:::my-s3-bucket/*" 
            ]
        }
    ]
}

-> Validate Policy でポリシーの書式チェックが出来ます
もし問題なかったら -> Create Policy でポリシー設定完了です。
ポリシーの一覧で、Filter: Customer Managed Polices を選択すると、作成したポリシーが出てきます。
あとは、Users作成をして、Groups設定でこのポリシーを選択すればOKです。

AWS CLIでアクセスできるかチェックしてみましょう。

AWS CLIのprofileを設定
$ aws configure --profile myprofile
AWS Access Key ID [None]: xx
AWS Secret Access Key [None]: xx
Default region name [None]: ap-northeast-1
Default output format [None]:
my-s3-bucket の中身を見る
$ aws s3 ls s3://my-s3-bucket --profile myprofile
2015-09-10 15:24:42          6 hello.txt
2015-09-10 15:10:09          2 test.txt
test.txtをアップロード
$ aws s3 cp test.txt s3://my-s3-bucket/test.txt --profile myprofile
対象外のバケツの中が見れない事を確認
$ aws s3 ls s3://this-is-test/ --profile myprofile
対象外のバケツにアップロードできないことを確認
$ aws s3 cp test.txt s3://this-is-test/test.txt --profile myprofile

以上で、S3のIPアドレス制限の設定が完了です。
注意事項としては、操作はできませんが、他のバケツ名も見えてしまう、というところがあります。これを制限する良い方法があったら、おしえていただけると嬉しいです。

Title pic from Providing Access to an IAM User in Another AWS Account That You Own - AWS Identity and Access Management
References
S3のアクセスコントロールまとめ - Qiita
Example Policies for Administering AWS Resources - AWS Identity and Access Management

Wednesday, September 9, 2015

Load Test of PHP require performance with PHP 5.6, PHP 7

Orange: H2O + PHP 7 (RC1)
RedH2O + PHP 5.6
Blue: Nginx + PHP 5.6

Spredsheet:
https://docs.google.com/spreadsheets/d/1XBqaxzJWtYQ1z28qHnUgoVOj2WVCypVeL0DaNQCkmcQ/edit?usp=sharing

Build Environment:
Amazon Linux AMI 2015.03.1 (HVM), SSD Volume Type - ami-1c1b9f1c
c4.xlarge + EBS (General Purpose SSD) 100GB

Load Test Description:
index.php has some require. and required file has only return ture.
# cat index.php
<?php
print 1;
require '/var/h2o/require1.php';
require '/var/h2o/require2.php';
require '/var/h2o/require3.php';
require '/var/h2o/require4.php';

# cat require1.php
<?php
return true;
Add require to index.php, make require file and do Apache Bench by this load test script.
https://gist.github.com/takeshiyako2/f8d8c6e1fc87dd6bd920

Note that:
Maybe, this test has noise, because I use cloud. so please confirm that with your own eyes.

Tuesday, September 8, 2015

cURL with HTTP/2


Do you want to check HTTP/2 with cURL command?
Yes. When you compile cURL with nghttp2 from source code, You can do it.

See olso.
HTTP/2 with curl
http://curl.haxx.se/docs/http2.html

Build Environment:
Amazon EC2
CentOS 7 (x86_64) with Updates HVM (ami-89634988)

Install some library
# yum -y groupinstall "Development Tools"
# yum -y install libev libev-devel zlib zlib-devel openssl openssl-devel git
Install nghttp2
# cd /var/tmp
# git clone https://github.com/tatsuhiro-t/nghttp2.git
# cd nghttp2
# autoreconf -i
# automake
# autoconf
# ./configure
# make
# make install
Set library search path
# echo '/usr/local/lib' > /etc/ld.so.conf.d/custom-libs.conf
# ldconfig
# ldconfig -p| grep libnghttp2
 libnghttp2.so.14 (libc6,x86-64) => /usr/local/lib/libnghttp2.so.14
 libnghttp2.so (libc6,x86-64) => /usr/local/lib/libnghttp2.so
Install cURL
# cd /var/tmp
# git clone https://github.com/bagder/curl.git
# cd curl
# ./buildconf
# ./configure --with-nghttp2=/usr/local

Check it.
  HTTP2 support:    enabled (nghttp2)

# make 
# make install
Check cURL Features. Does it have HTTP2?
# ./src/curl -V
curl 7.45.0-DEV (x86_64-unknown-linux-gnu) libcurl/7.45.0-DEV OpenSSL/1.0.1e zlib/1.2.7 nghttp2/1.3.1-DEV
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: IPv6 Largefile NTLM NTLM_WB SSL libz HTTP2 UnixSockets
Check HTTP/2 response with Test URL.
# ./src/curl --http2 -v https://http2bin.org/get

*   Trying 104.131.161.90...
* Connected to http2bin.org (104.131.161.90) port 443 (#0)
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* NPN, negotiated HTTP2 (h2)
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Unknown (67):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* Server certificate:
*   subject: OU=Domain Control Validated; OU=EssentialSSL; CN=www.http2bin.org
*   start date: 2015-02-21 00:00:00 GMT
*   expire date: 2016-02-21 23:59:59 GMT
*   subjectAltName: http2bin.org matched
*   issuer: C=GB; ST=Greater Manchester; L=Salford; O=COMODO CA Limited; CN=COMODO RSA Domain Validation Secure Server CA
*   SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x1c59450)
> GET /get HTTP/1.1
> Host: http2bin.org
> User-Agent: curl/7.45.0-DEV
> Accept: */*
>
* http2_recv: 16384 bytes buffer at 0x1c59d60 (stream 1)
* http2_recv: 16384 bytes buffer at 0x1c59d60 (stream 1)
* http2_recv: 16384 bytes buffer at 0x1c59d60 (stream 1)
* http2_recv: returns 466 for stream 1
< HTTP/2.0 200
< server:h2o/1.2.0
< date:Tue, 08 Sep 2015 03:41:11 GMT
< content-type:application/json
< access-control-allow-origin:*
< access-control-allow-credentials:true
< x-clacks-overhead:GNU Terry Pratchett
<
{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Connection": "keep-alive",
    "Host": "http2bin.org",
    "User-Agent": "curl/7.45.0-DEV",
    "Via": "2 http2bin.org"
  },
  "origin": "54.65.241.177",
  "url": "https://http2bin.org/get"
}
* Connection #0 to host http2bin.org left intact



Titile pic from Duncan Hull The Internet: Yours For Just £1

Monday, September 7, 2015

Install H2O with PHP7 (RC1)


This is How to Install H2O with PHP7 (RC1). 

Build Environment:
Amazon Linux AMI 2015.03.1 (HVM), SSD Volume Type - ami-1c1b9f1c
c4.xlarge + EBS (General Purpose SSD) 100GB

Install H2O
See Install H2O with PHP5.6.

Install some librarys
# yum -y install libxml2-devel bison bison-devel openssl-devel curl-devel libjpeg-devel libpng-devel libmcrypt-devel readline-devel libtidy-devel libxslt-devel re2c
Install phpenv
# cd /var/tmp
# git clone https://github.com/CHH/phpenv.git
# cd phpenv/bin/
# sh phpenv-install.sh
# export PATH="/root/.phpenv/bin:$PATH"
# eval "$(phpenv init -)"
# source ~/.bashrc
Install php-build & PHP7
# cd /var/tmp
# git clone https://github.com/CHH/php-build.git
# cd php-build/
# sh install.sh
# ./bin/php-build --definitions | grep 7.0
7.0.0alpha1
7.0.0alpha2
7.0.0beta1
7.0.0beta2
7.0.0beta3
7.0.0RC1
# ./bin/php-build 7.0.0RC1 ~/.phpenv/versions/7.0.0RC1
# ~/.phpenv/versions/7.0.0RC1/bin/php -v
PHP 7.0.0RC1 (cli) (built: Sep  4 2015 09:08:33)
Copyright (c) 1997-2015 The PHP Group
Zend Engine v3.0.0-dev, Copyright (c) 1998-2015 Zend Technologies
    with Zend OPcache v7.0.6-dev, Copyright (c) 1999-2015, by Zend Technologies
Set confilg file
# cp /root/.phpenv/versions/7.0.0RC1/etc/php-fpm.conf.default /root/.phpenv/versions/7.0.0RC1/etc/php-fpm.conf
# cp /root/.phpenv/versions/7.0.0RC1/etc/php-fpm.d/www.conf.default /root/.phpenv/versions/7.0.0RC1/etc/php-fpm.d/www.conf
Run php-fpm
# ~/.phpenv/versions/7.0.0RC1/sbin/php-fpm &
# ps auxf | grep php | grep -v grep
root      2660  0.0  0.0 296828  6240 ?        Ss   09:23   0:00 php-fpm: master process (/root/.phpenv/versions/7.0.0RC1/etc/php-fpm.conf)
nobody    2661  0.0  0.0 296828  5824 ?        S    09:23   0:00  \_ php-fpm: pool www
nobody    2662  0.0  0.0 296828  5828 ?        S    09:23   0:00  \_ php-fpm: pool www
Check
# curl -v http://localhost/
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 80 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.40.0
> Host: localhost
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Fri, 04 Sep 2015 09:25:07 GMT
< Server: h2o/1.5.0alpha-1
< Connection: keep-alive
< x-powered-by: PHP/7.0.0RC1
< content-type: text/html; charset=UTF-8
< transfer-encoding: chunked
<
* Connection #0 to host localhost left intact
1
Do Apache Bench
# ab -c 10 -n 10000 http://localhost/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests


Server Software:        h2o/1.5.0alpha-1
Server Hostname:        localhost
Server Port:            80

Document Path:          /
Document Length:        1 bytes

Concurrency Level:      10
Time taken for tests:   0.546 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      1700170 bytes
HTML transferred:       10001 bytes
Requests per second:    18300.54 [#/sec] (mean)
Time per request:       0.546 [ms] (mean)
Time per request:       0.055 [ms] (mean, across all concurrent requests)
Transfer rate:          3038.48 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:     0    0   0.5      0      15
Waiting:        0    0   0.5      0      15
Total:          0    1   0.5      0      15
ERROR: The median and mean for the total time are more than twice the standard
       deviation apart. These results are NOT reliable.

Percentage of the requests served within a certain time (ms)
  50%      0
  66%      1
  75%      1
  80%      1
  90%      1
  95%      1
  98%      1
  99%      1
 100%     15 (longest request)


Titile pic from Eli White

Friday, September 4, 2015

Install H2O with PHP5.6

yonolatengo H2O......Go https://www.flickr.com/photos/yonolatengo/4905865998/in/photolist-8tvR3w-6wtbBX-6wxmgq-6wxkNu-6wt8vi-6wt7Dt-6wt6vH-6wt9Y8-bqokYN-2oyZAS-5kM5Pe-awZwAu-4QCzLp-6qKPtC-gr1GCZ-rDLrRB-5ng3vh-6L3UUo-2Tfcs-7XN2zL-bocyBY-5K1Ti5-4es3uw-e1ZPVx-4es2Ej-7oPMdP-7KX3qi-8W2EYo-8PADpe-8W2GdC-8VYBE4-8W2EhY-8W2DFL-8VYzae-8VYysX-8VYwu2-6wt73T-72ZuTT-8c3PzC-awWQzc-awWQ9P-25Eqqb-a3EUjj-6FjkA-oqu4AT-7GBs3i-4eo3UV-awWPs4-25EpAS-4eo4JP
https://h2o.examp1e.net/

H2O is known as HTTP(HTTP/2) Server of new era.
This is simple install log for H2O with php-fpm.

Build Environment:
Amazon Linux AMI 2015.09 (HVM), SSD Volume Type - ami-9a2fb89a
c4.xlarge + EBS (General Purpose SSD) 100GB

Install some librarys
# yum -y groupinstall "Development Tools"
# yum -y install libyaml-devel openssl-devel cmake httpd-tools php56-fpm php56-opcache
Install libuv
# cd /var/tmp
# git clone https://github.com/libuv/libuv.git
# cd libuv
# ./autogen.sh
# ./configure --prefix=/usr/local/libuv
# make -j 4
# make install
Install H2O
# cd /var/tmp
# git clone https://github.com/h2o/h2o.git
# cd h2o/
# PKG_CONFIG_PATH=/usr/local/libuv/lib/pkgconfig/ cmake -DWITH_BUNDLED_SSL=on -DCMAKE_INSTALL_PREFIX=/usr/local/h2o .
# make -j 4
# make install
# ln -s /usr/local/h2o/bin/h2o /usr/bin/h2o
# h2o -v
h2o version 1.5.0
OpenSSL: LibreSSL 2.2.2
mruby: YES
Set config file for h2o with php-fpm
# emacs /etc/h2o.conf
pid-file: /var/run/h2o.pid
user: nobody
error-log: /var/log/h2o.error.log
access-log: /var/log/h2o.access.log
listen: 80

# php-fpm
file.custom-handler:
  extension: .php
  fastcgi.connect:
    host: 127.0.0.1
    port: 9000
    type: tcp

# Directory Index
file.index: [ 'index.php', 'index.html' ]

# Hosts
hosts:
  "localhost:80":
    paths:
      /:
        # www root
        file.dir: /var/h2o
        file.dirlisting: on

Make test php file
# mkdir /var/h2o/
# echo '<?php print 1;' > /var/h2o/index.php
Start php-fpm
# service php-fpm start
Start H2O
# h2o -m daemon -c /etc/h2o.conf
Check
# curl -v http://localhost/
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 80 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.40.0
> Host: localhost
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Fri, 02 Oct 2015 09:33:44 GMT
< Server: h2o/1.5.0
< Connection: keep-alive
< x-powered-by: PHP/5.6.13
< content-type: text/html; charset=UTF-8
< transfer-encoding: chunked
<
* Connection #0 to host localhost left intact
1
Do Apache Bench
# ab -c 10 -n 10000 http://localhost/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests


Server Software:        h2o/1.5.0
Server Hostname:        localhost
Server Port:            80

Document Path:          /
Document Length:        1 bytes

Concurrency Level:      10
Time taken for tests:   0.644 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      1610000 bytes
HTML transferred:       10000 bytes
Requests per second:    15521.42 [#/sec] (mean)
Time per request:       0.644 [ms] (mean)
Time per request:       0.064 [ms] (mean, across all concurrent requests)
Transfer rate:          2440.38 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       0
Processing:     0    1   0.3      0       3
Waiting:        0    0   0.3      0       3
Total:          0    1   0.3      1       3
ERROR: The median and mean for the processing time are more than twice the standard
       deviation apart. These results are NOT reliable.

Percentage of the requests served within a certain time (ms)
  50%      1
  66%      1
  75%      1
  80%      1
  90%      1
  95%      1
  98%      1
  99%      2
 100%      3 (longest request)
Stop H2O
# kill -TERM `cat /var/run/h2o.pid`


Title pic from yonolatengo H2O......Go