2015年12月21日月曜日

DockerでLet's EncryptのSSLを導入する

無料でSSL証明書を自動発行するサービス Let's Encrypt を利用して、nginxにSSL証明書を組み込む手順を紹介します。
通常、Let's Encryptの証明書を発行するには、Pythonやモジュールをインストールして、Lets's Encrypt Clientを使えるようにする必要がありますが、これらの環境構築を簡素化するためにDockerを利用します。
以下導入手順です。OSはCentOS 6を使用しています。

Dockerをインストール
# yum install docker-io
# service docker start
# service docker status
docker (pid  2257) is running...
※ もし。Dockerが起動できず。以下のようなメッセージが出るときは、カーネルのバージョンを上げる必要があります。
# cat /var/log/docker
\nWed Aug 17 15:27:36 JST 2016\n
time="2016-08-17T15:27:36.884624234+09:00" level=warning msg="You are running linux kernel version 2.6.32-431.17.1.el6.x86_64, which might be unstable running docker. Please upgrade your kernel to 3.10.0."
/usr/bin/docker: relocation error: /usr/bin/docker: symbol dm_task_get_info_with_deferred_remove, version Base not defined in file libdevmapper.so.1.02 with link time reference
yumで最新のカーネルをインストールして、サーバをリブートします。
# yum clean all
# yum -y upgrade

Let's Encryptで証明書を発行
コマンドを実行すると、--webroot-path で指定したパス以下に、.well-known/acme-challenge/* ディレクトリを作って認証ファイルが設置されます。
Let's Encryptの認証クローラが、このファイルにアクセスして認証処理をします。
Dockerコンテナから、外部ファイルへアクセスする必要があるので、 -v オプションに --webroot-path と同じパスを指定しておきます。

以下の例では、--webroot-path に /usr/share/nginx/html を指定。
同じく、-v "/usr/share/nginx/html":"/usr/share/nginx/html"を指定。
-d にドメイン historia.hitokoto.co を指定。
-m にメールアドレスを指定します。
# docker run -it --rm --name letsencrypt \
            -v "/etc/letsencrypt:/etc/letsencrypt" \
            -v "/var/lib/letsencrypt:/var/lib/letsencrypt" \
            -v "/usr/share/nginx/html":"/usr/share/nginx/html" \
            quay.io/letsencrypt/letsencrypt:latest certonly \
            --agree-tos \
            --webroot \
            --webroot-path /usr/share/nginx/html \
            -d historia.hitokoto.co \
            -m yako.takeshi@googlemail.com
IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at
   /etc/letsencrypt/live/historia.hitokoto.co/fullchain.pem. Your cert
   will expire on 2016-03-20. To obtain a new version of the
   certificate in the future, simply run Let's Encrypt again.
 - If you like Let's Encrypt, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le
証明書ができた旨が示されます。
この例では、/etc/letsencrypt/live/historia.hitokoto.co/ 以下にファイルができています。 確認してみましょう。
# ls -al /etc/letsencrypt/live/historia.hitokoto.co
total 8
drwxr-xr-x 2 root root 4096 Dec 21 13:48 .
drwx------ 3 root root 4096 Dec 21 13:48 ..
lrwxrwxrwx 1 root root   44 Dec 21 13:48 cert.pem -> ../../archive/historia.hitokoto.co/cert1.pem
lrwxrwxrwx 1 root root   45 Dec 21 13:48 chain.pem -> ../../archive/historia.hitokoto.co/chain1.pem
lrwxrwxrwx 1 root root   49 Dec 21 13:48 fullchain.pem -> ../../archive/historia.hitokoto.co/fullchain1.pem
lrwxrwxrwx 1 root root   47 Dec 21 13:48 privkey.pem -> ../../archive/historia.hitokoto.co/privkey1.pem

OpenSSLのサーバコマンドで証明書をチェック
s_server を起動。オプションで証明書を指定します。さらに、-wwwオプションを付けて HTTPS サーバをエミュレートします。
# cd /etc/letsencrypt/live/historia.hitokoto.co
# openssl s_server -cert cert.pem -key privkey.pem -CAfile chain.pem -www
Using default temp DH parameters
Using default temp ECDH parameters
opensslのs_serverがポート 4433 番で接続待ちをしている状態になりました。

クライアントを起動。
$ openssl s_client
CONNECTED(00000003)
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X1
verify return:1
depth=0 CN = historia.hitokoto.co
verify return:1
---
Certificate chain
 0 s:/CN=historia.hitokoto.co
   i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X1
 1 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X1
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3
---
Server certificate
以下略
ハンドシェイクの様子が流れて、入力待ちとなります。

Curlで確認してみます。
# curl https://historia.hitokoto.co:4433/ -H 'Host: historia.hitokoto.co' -v
* About to connect() to historia.hitokoto.co port 4433 (#0)
*   Trying 153.126.157.62... connected
* Connected to historia.hitokoto.co (153.126.157.62) port 4433 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_DHE_RSA_WITH_AES_256_CBC_SHA
* Server certificate:
*  subject: CN=historia.hitokoto.co
*  start date: Dec 21 03:49:00 2015 GMT
*  expire date: Mar 20 03:49:00 2016 GMT
*  common name: historia.hitokoto.co
*  issuer: CN=Let's Encrypt Authority X1,O=Let's Encrypt,C=US
> GET / HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.19.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Accept: */*
> Host: historia.hitokoto.co
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200 ok
< Content-type: text/html
<
以下略
SSLで通信ができていることが確認できました。

nginxにLet's Encryptの証明書を組み込む
DH鍵交換に使用するパラメータファイルを作成。
# openssl dhparam 2048 -out dhparam.pem
nginx設定ファイルを編集。serverディレクティブを以下のようにしました。
nginxをリスタート。
# service nginx restart
Stopping nginx:                                            [  OK  ]
Starting nginx:

Qualys SSL LabsでSSLの評価を試してみます。
https://www.ssllabs.com/ssltest/index.html

A+となりました。

SSL証明書を更新
証明書を発行する時とほぼ同じコマンドです。--renew-by-defaultオプションを付けています。
状況によって、--interactive=false --tty=false オプションも付けます。
# docker run --rm --name letsencrypt \
            -v "/etc/letsencrypt:/etc/letsencrypt" \
            -v "/var/lib/letsencrypt:/var/lib/letsencrypt" \
            -v "/usr/share/nginx/html":"/usr/share/nginx/html" \
            quay.io/letsencrypt/letsencrypt:latest certonly \
            --webroot \
            --webroot-path /usr/share/nginx/html \
            -d historia.hitokoto.co \
            --renew-by-default
証明書の期限が90日と短いです。そのため、cronなどでSSL証明書を自動更新するようにしておくと良いでしょう。

See Also.
Title image from Let's Encrypt - Overview
User Guide — Let's Encrypt 0.2.0.dev0 documentation
https://letsencrypt.readthedocs.org/en/latest/using.html#running-with-docker
Let's Encrypt を支える ACME プロトコル - Block Rockin’ Codes
http://jxck.hatenablog.com/entry/letsencrypt-acme
Linux - Let's Encrypt で証明書を取得した時の手順備忘録 - Qiita
http://qiita.com/TsutomuNakamura/items/4166423699061e38d296
letsencrypt - DockerでLet’s Encryptしよっか - Qiita
http://qiita.com/sawanoboly/items/9fdde1707de5e975dd15
Let's EncryptのSSL証明書で、Qualys SSLTestでA+評価を獲得するには - Qiita
http://qiita.com/dseg/items/bab80f6f14349fcd9c22
Dockerを導入する(CentOS版) - Qiita
http://qiita.com/zwirky/items/991f61a231f4e198a320

2015年12月20日日曜日

Android開発 AdMobを実装して、インタースティシャル広告を配信する

AdMobを使って、インタースティシャル広告を出してみます。いわゆる全画面広告です。
※AdMobの登録方法などは、前回の記事を参考にしてください。 -> Android開発 AdMobを実装して、バナー広告を配信する
今回は、バックボタンで画面を閉じるときにインタースティシャル広告を出すようにしました。
さらに、毎回、広告が出るとユーザとしてうざいのでは?と考えて、数回に1回だけ広告を出すようにしています。
以下、サンプルコードです。
Empty Activityで作りました。


See Also.
インタースティシャル広告 | AdMob Android ガイド | Google Developers
https://developers.google.com/mobile-ads-sdk/docs/admob/android/interstitial?hl=ja

2015年12月19日土曜日

Android開発 AdMobを実装して、バナー広告を配信する

AdMobのバナー広告を表示する方法を書いていきます。

AdMobは、Googleが提供しているモバイルアプリ向けの広告サービスです。モバイルアプリ向けの広告サービスはいくつかの会社が提供しているのですが、もっとも導入が手軽なAdMobを扱ってみます。
AdMob以外にも、いろいろな広告会社のアプリ向け広告サービスがありますが、AdMobが他の広告会社より有利に立っている点といえば、AndroidとGoogleアカウントをひも付いていることによる、ターゲティング配信です。バナー広告といったスタンダードな枠では、ほかの追随を許さない収益性を得ることができる場合が多いです。
もし、アクティブユーザが多いアプリであるならば、ほかの会社のサービスを試してみても良いでしょう。AdMobは広告枠の種類が少ないので、より多くの広告のサイズや種類を選ぶことが出来るはずです。アプリのデザインにあった広告を出すことが大事だと思います。最近は、インタースティシャルの動画広告が人気出てるみたいですね。

さっそく実装してみましょう。
AdMobのアカウントを作っておきます。
https://apps.admob.com/

AdMobにアプリを登録して、広告ユニットIDをメモしておきます。
ca-app-pub-XXXXXXXXXXXXXXXX/NNNNNNNNNN のような文字列です。下の実装では、テスト用の広告ユニットIDを使用します。Google Playにアップロードするときに、自分の広告ユニットIDに書き換えておきましょう。

以下、AdMobバナー広告の実装方法です。
Empty Activityで作ります。
以下コードサンプルです。

build.gradle に compile 'com.google.android.gms:play-services:6.+' を追記。

AndroidManifest にいくつか追記。

strings.xml に広告ユニット ID を設定。テスト用の広告ユニットIDです。

レイアウトに AdView を配置。
広告に使用する追加の名前空間 http://schemas.android.com/apk/res-auto を追記、さらに、AdView 用の要素を追記。

MainActivity クラスに広告を設置。
以下のクラスをインポートしておきます。
import com.google.android.gms.ads.AdRequest;
import com.google.android.gms.ads.AdView;
AdView を検索するコードを追加。AdRequest を作成して AdView に広告を読み込みます。
ビルドしてみましょう。
バナー広告が表示されたでしょうか?

See Also
Android クイック スタート | AdMob Android ガイド | Google Developers
https://developers.google.com/mobile-ads-sdk/docs/admob/android/quick-start?hl=ja

2015年12月8日火曜日

yum install php7 on CentOS


Let's isntall PHP 7 to CentOS by yum.

This is a CentOS 6.7.
# cat /etc/redhat-release
CentOS release 6.7 (Final)
Install remi repo, some librarys and PHP 7 with some extensions.
# rpm -ivh http://rpms.famillecollet.com/enterprise/remi-release-6.rpm
# yum install libxslt libxslt-devel postgresql-devel httpd-devel autoconf automake gd-devel
# yum install --disablerepo=base,updates --enablerepo=remi --enablerepo=remi-php70 php php-common php-mbstring php-pdo php-cli php-devel php-fpm php-mysqlnd php-gd php-bcmath php-pecl-geoip php-pgsql php-pecl-memcached php-pecl-uuid php-pear php-opcache
Show version of PHP. It's PHP 7.
# php -v
PHP 7.0.0 (cli) (built: Dec  5 2015 07:17:51) ( NTS )
Copyright (c) 1997-2015 The PHP Group
Zend Engine v3.0.0, Copyright (c) 1998-2015 Zend Technologies
    with Zend OPcache v7.0.6-dev, Copyright (c) 1999-2015, by Zend Technologies
Fine! That's it.

Top image from Ian Muttoo yum yum too

2015年12月4日金曜日

Go lang bbpd / Amazon DynamoDBのプロキシサーバ

Go Advent Calendar 2015 - Qiita 11/4の記事です。

Go言語で書かれた、Amazon DynamoDBのプロキシサーバの紹介をします。
もともと、他言語のPHPで書かれたウェブアプリケーションで、AWS SDK for PHPを利用してDynamoDBを使おうとしていたのですが、Class Preloader for PHPがうまく動かせず、全然パフォーマンスが出なくて困っていました。EC2 c4.xlarge nginx + php-fpm でベンチマークして800 req/s ほどでした。(PHP with Amazon DynamoDB
そのため、他言語でDynamoDBのプロキシサーバを立てて、そのAPIを経由してデータのSet / Getをしようと考えて、色々探しまわった結果、SmugMugという企業がオープンソースで公開している、Goで書かれた bbpd というソフトウェアを見つけました。これを試してみたところ、思いのほか簡単にパフォーマンスの良いプロキシサーバを建てることができたので、紹介したいと思います。同じように困っている方を想定して、Goをインストールするところから書いていきます。

環境: EC2 Instance Type: c4.xlarge / AMI ID: Amazon Linux AMI 2015.09.1 (HVM), SSD Volume Type - ami-383c1956

Goをインストール
$ sudo yum -y install golang zsh
$ go version
go version go1.4.2 linux/amd64
$ mkdir ~/gocode
$ echo 'export GOPATH=$HOME/gocode' >> ~/.bashrc
$ source $HOME/.bashrc

bbpdをインストール
$ go get github.com/smugmug/bbpd
$ sudo cp $GOPATH/bin/bbpd /usr/bin/
$ sudo cp $GOPATH/src/github.com/smugmug/bbpd/bin/bbpd/bbpd_ctl /usr/bin/
$ sudo cp $GOPATH/src/github.com/smugmug/bbpd/bin/bbpd/bbpd_daemon /usr/bin/
$ sudo chmod 755 /usr/bin/bbpd*
設定ファイルを編集。
アクセスキー、シークレットキーを入力。Tokyoリュージョンの場合は、us-east-1をap-northeast-1に変更。
$ cp $GOPATH/src/github.com/smugmug/godynamo/conf_file/test_aws-config.json ~/.aws-config.json
$ emacs ~/.aws-config.json
{
    "extends":[],
    "services": {
        "default_settings":{
            "params":{
                "access_key_id":"myAccessKey",
                "secret_access_key":"mySecret",
                "use_sys_log":true
            }
        },
        "dynamo_db": {
            "host":"dynamodb.us-east-1.amazonaws.com",
            "zone":"us-east-1",
            "scheme":"http",
            "port":80,
            "keepalive":false,
            "iam": {
                "use_iam":false,
                "role_provider":"",
                "access_key":"",
                "secret_key":"",
                "token":"",
                "base_dir":"",
                "watch":false
            }
        }
    }
}
bbpdデーモンをスタート。デフォルトでは、ポート番号12333と12334の2つを受け付けるようになります。
$ bbpd_ctl start
**** starting bbpd as daemon
bbpd - started
2015/11/10 07:40:12 global conf.Vals initialized
2015/11/10 07:40:12 not using iam, assume credentials hardcoded in conf file
2015/11/10 07:40:12 starting bbpd...
2015/11/10 07:40:12 induce panic with ctrl-c (kill -2 2888) or graceful termination with kill -[1,3,15] 2888
2015/11/10 07:40:12 trying to bind to port:12333
2015/11/10 07:40:12 init routing on port 12333
2015/11/10 07:40:12 global conf.Vals initialized
2015/11/10 07:40:12 not using iam, assume credentials hardcoded in conf file
2015/11/10 07:40:12 starting bbpd...
2015/11/10 07:40:12 induce panic with ctrl-c (kill -2 2894) or graceful termination with kill -[1,3,15] 2894
2015/11/10 07:40:12 trying to bind to port:12333
2015/11/10 07:40:12 port 12333 already in use
2015/11/10 07:40:12 trying to bind to port:12334
2015/11/10 07:40:12 init routing on port 12334
プロセスを確認。
$ pgrep -l bbpd
2888 bbpd
2894 bbpd
CurlでAPIを叩いてステータスを確認。
$ curl -H "X-Bbpd-Indent: true" "http://localhost:12333/Status"
{
 "Status": "ready",
 "AvailableHandlers": [
  "/DescribeTable/",
  "/DeleteItem",
  "/ListTables",
  "/CreateTable",
  "/UpdateTable",
  "/StatusTable/",
  "/PutItem",
  "/PutItemJSON",
  "/GetItem",
  "/GetItemJSON",
  "/BatchGetItem",
  "/BatchGetItemJSON",
  "/BatchWriteItem",
  "/BatchWriteItemJSON",
  "/UpdateItem",
  "/Query",
  "/Scan",
  "/RawPost/",
  "/"
 ],
 "Args": {
  "X-Bbpd-Indent": "set '-H \"X-Bbpd-Indent: True\" ' to indent the top-level json",
  "X-Bbpd-Verbose": "set '-H \"X-Bbpd-Verbose: True\" ' to get verbose output"
 },
 "Summary": {
  "StartTime": "2015-11-10 07:40:12.993368854 +0000 UTC",
  "RunningTime": "1m17.069634284s",
  "LongestResponse": "0.00ms",
  "AverageResponse": "0.00ms",
  "LastResponse": "no requests made yet",
  "ResponseCount": "0"
 }
}
bbpdが立ち上がりました。

テーブルを作成
$ curl -H "X-Amz-Target: DynamoDB_20120810.CreateTable" -X POST -d '
{
    "AttributeDefinitions": [
        {
            "AttributeName": "user_id", 
            "AttributeType": "S"
        }
    ], 
    "KeySchema": [
        {
            "AttributeName": "user_id", 
            "KeyType": "HASH"
        }
    ], 
    "ProvisionedThroughput": {
        "ReadCapacityUnits": 100, 
        "WriteCapacityUnits": 100
    }, 
    "TableName": "bbpd-test"
}
' http://localhost:12333/
{"TableDescription":{"AttributeDefinitions":[{"AttributeName":"user_id","AttributeType":"S"}],"CreationDateTime":1.447141682984E9,"ItemCount":0,"KeySchema":[{"AttributeName":"user_id","KeyType":"HASH"}],"ProvisionedThroughput":{"NumberOfDecreasesToday":0,"ReadCapacityUnits":100,"WriteCapacityUnits":100},"TableArn":"arn:aws:dynamodb:ap-northeast-1:891959351900:table/bbpd-test","TableName":"bbpd-test","TableSizeBytes":0,"TableStatus":"CREATING"}}

Put
$ curl -H "X-Amz-Target: DynamoDB_20120810.PutItem"  -X POST -d '
{
    "Item": {
        "num": 1, 
        "numlist": [
            6, 
            7, 
            1, 
            2, 
            3, 
            9, 
            -7234234234.234234
        ], 
        "stringlist": [
            "pk1_a", 
            "pk1_b", 
            "pk1_c"
        ], 
        "user_id": "my_user_id1"
    }, 
    "TableName": "bbpd-test"
}
' http://localhost:12333/PutItemJSON

Get
$ curl -H "X-Amz-Target: DynamoDB_20120810.GetItem" -X POST -d '
{
    "Key": {
        "user_id": {
            "S": "my_user_id1"
        }
    }, 
    "TableName": "bbpd-test"
}
' "http://localhost:12333/GetItemJSON"
{"Item":{"num":1,"numlist":[6,3,9,7,2,1,-7.234234234234234e+09],"stringlist":["pk1_b","pk1_c","pk1_a"],"user_id":"my_user_id1"}
うまく動いていますね!

ベンチマーク
DynamoDB: Provisioned Read Capacity Units: 10000 / Provisioned Write Capacity Units: 10000
EC2: Instance type: c4.large, c4.xlarge, c4.2xlarge

Getのパフォーマンスを計測。
# emacs postdata.file
{
    "Key": {
        "user_id": {
            "S": "my_user_id1"
        }
    },
    "TableName": "bbpd-test"
}
# ab -n 100000 -c 100 -p postdata.file -T "application/json" http://localhost:12333/GetItemJSON

c4.large
Requests per second:    4754.25 [#/sec] (mean)

c4.xlarge
Requests per second:    8591.35 [#/sec] (mean)

c4.2xlarge
Requests per second:    8540.63 [#/sec] (mean) 
8000 req/s くらいあれば、殆どのウェブアプリケーションでは、要件を満たせるのではないでしょうか。

注意
更に、コネクション数を増やしていくと、DynamoDBのキャパシティを超えて、APIから400エラーが返ってきます。その場合は下記のコードでリトライ処理。
https://github.com/smugmug/godynamo/blob/master/authreq/authreq.go#L152
デフォルトでは、7回までリトライします。もし、"authreq.retryReq: failed retries on %s:%s"などといったログ出力されていなければ、リクエストが成功していますが。念のため、ログの出力をモニタリングしておくと良いでしょう。
ほかに、ファイルディスクリプタ上限も上げておくと良いです。

PHPからbbpdを使ってみる
PHPからbbpdを叩いたときのベンチマークです。phpのコードはこちら
Instance type: c4.xlarge
# ab -c 100 -n 10000 http://127.0.0.1/get_go.php
Requests per second:    4260.64 [#/sec] (mean)
AWS SDK for PHPを使った時は、800req/sくらいだったので、5倍パフォーマンスが良いです。これで、要件によっては、その範疇に収まるようにできるのではないでしょうか。選択肢のひとつとして考えてみるのもありかと思います。

GoのコードでGoDynamoを使ってみる
GoからGoDynamoを使うコードはこのような感じになります。プロダクション用に書くならば、テストコード(https://github.com/smugmug/godynamo/tree/master/tests)を参考にすると良いと思います。

まとめ
bbpd・GoDynamoを使ってDynamoDBにアクセスしてみました。
GoDynamoの内部では、公式のAWS SDKと同じく、自動再試行ロジックを実装するなど、丁寧に作られている印象があって好感が持てました。とくに、bbpdに関しては、必要な機能だけを持たせたAPIサーバということで、マイクロサービス感がありました。今回は、PHPからbbpdを叩いてDynamoDBにアクセスするといった使い方をしていて、個人的にはAWS SDKのコードを修正したり、PHP ExtensionをCで頑張って書いたりするよりは楽だったのが良かったところです。

See Also.
SmugMug: From MySQL to Amazon DynamoDB (DAT204) | AWS re:Invent 2013 http://www.slideshare.net/AmazonWebServices/smugmug-from-mysql-to-amazon-dynamodb-dat204-aws-reinvent-2013
Handling Errors in DynamoDB Operations - Amazon DynamoDB https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/ErrorHandling.html
GoDynamo
https://github.com/smugmug/godynamo
AWS SDK for Go - General Availability
https://aws.amazon.com/about-aws/whats-new/2015/11/aws-sdk-for-go-general-availability/
Amazon DynamoDB logo / Amazon Web Services LLC - http://aws.typepad.com/aws/2011/12/introducing-aws-simple-icons-for-your-architecture-diagrams.html
gopher-side_color.{ai,svg,png} was created by Takuya Ueda (http://u.hinoichi.net). Licensed under the Creative Commons 3.0 Attributions license. - https://github.com/golang-samples/gopher-vector#gopher-side_color
The Go gopher was designed by Renee French. (http://reneefrench.blogspot.com/)

2015年11月9日月曜日

The first step of Norikra

http://norikra.github.io/
https://github.com/norikra/norikra
Norikra is an open-source Stream Processing Server with SQL.
Schema-less event streams (called as 'target')
SQL processing with window specifier supports, and JOINs, SubQueries
Complex input/output events with nested Hashes and Arrays, and Query supports
Dynamic query registration/removing, without any restarts
Ultra fast bootstrap and small start
UDF plugins
This entry provides that how to run Norikra on Docker, and some example of put JSON and get result by SQL like query over HTTP API. Why using Norikra on Docker? I thought that It is important to cut down on install operations as much as possible.
Let's try.

Environment: CentOS 6.7

Install Norikra on Docker
Install Docker.
# yum install docker-io
# service docker start
Run Norikra on Docker. Use this Docker image. https://registry.hub.docker.com/u/myfinder/docker-norikra/
# docker pull myfinder/docker-norikra
# docker run -d -p 26578:26578 -p 26571:26571 -v /var/tmp/norikra:/var/tmp/norikra:rw -t myfinder/docker-norikra norikra start --stats /var/tmp/norikra/stats.json -l /var/tmp/norikra
Check it. What's up? Norikra.
# docker ps
CONTAINER ID        IMAGE                     COMMAND                CREATED             STATUS              PORTS                                                NAMES
25842cda847f        myfinder/docker-norikra   "norikra start --sta   5 seconds ago       Up 4 seconds        0.0.0.0:26571->26571/tcp, 0.0.0.0:26578->26578/tcp   naughty_mcclintock
# docker exec -it 25842cda847f ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0 10 03:07 ?        00:00:28 java -Xmx500m -Xss2048k -Djffi.boot.library.path=/jruby-1.7.19/lib/jni -server -XX:-UseGCO
root        64     0  0 03:11 ?        00:00:00 ps -ef
About one minute later..
# tail -f /var/tmp/norikra/norikra.log
2015-11-09 03:08:16 +0000 [INFO] : thread configurations, engine:{:inbound=>{:threads=>0, :capacity=>0}, :outbound=>{:threads=>0, :capacity=>0}, :route_exec=>{:threads=>0, :capacity=>0}, :timer_exec=>{:threads=>0, :capacity=>0}}, rpc:{:threads=>2}, web:{:threads=>2}
2015-11-09 03:08:16 +0000 [INFO] : logging configurations, level:nil, dir:"/var/tmp/norikra", filesize:nil, backups:nil, bufferlines:nil
2015-11-09 03:08:17 +0000 [INFO] : Loading UDF plugins
2015-11-09 03:08:17 +0000 [INFO] : Loading Listener plugins
2015-11-09 03:08:17 +0000 [INFO] : Listener loaded, name:Norikra::Listener::Stdout
2015-11-09 03:08:17 +0000 [INFO] : Listener loaded, name:Norikra::Listener::Loopback
2015-11-09 03:08:17 +0000 [INFO] : RPC server 0.0.0.0:26571, 2 threads
2015-11-09 03:08:17 +0000 [INFO] : WebUI server 0.0.0.0:26578, 2 threads
2015-11-09 03:08:17 +0000 [INFO] : Norikra server started.
# curl -I localhost:26578
HTTP/1.1 200 OK
You can see web console.

Examples
Set alias of norikra-client for docker exec.
# alias norikra-client='docker exec -it 25842cda847f norikra-client'
Make target.
# norikra-client target open www path:string status:integer referer:string agent:string userid:integer
# norikra-client target list
TARGET AUTO_FIELD
www true
1 targets found.

Example 1
Make querie name of "www.toppageviews".
# norikra-client query add www.toppageviews 'SELECT count(*) AS cnt FROM www.win:time_batch(10 sec) WHERE path="/" AND status=200'
Send events.
# curl -X POST -H "Content-Type: application/json" --data '{"target":"www", "events":[{"path":"/", "status":200, "referer":"", "agent":"MSIE", "userid":3}]}' http://localhost:26578/api/send
# curl -X POST -H "Content-Type: application/json" --data '{"target":"www", "events":[{"path":"/login", "status":301, "referer":"/", "agent":"MSIE", "userid":3}]}' http://localhost:26578/api/send
# curl -X POST -H "Content-Type: application/json" --data '{"target":"www", "events":[{"path":"/content", "status":200, "referer":"/login", "agent":"MSIE", "userid":3}]}' http://localhost:26578/api/send
# curl -X POST -H "Content-Type: application/json" --data '{"target":"www", "events":[{"path":"/page/1", "status":200, "referer":"/content", "agent":"MSIE", "userid":3}]}' http://localhost:26578/api/send
Result of querie.
# curl -X GET -H "Content-Type: application/json" --data '{"query_name":"www.toppageviews"}' http://localhost:26578/api/see
[[1447046244,{"cnt":1}]]

Example 2
Add other querie name of "www.search".
# norikra-client query add www.search 'SELECT count(*) AS cnt FROM www.win:time_batch(10 sec) WHERE path="/content" AND search_param.length() > 0'
Send events.
# curl -X POST -H "Content-Type: application/json" --data '{"target":"www", "events":[{"path":"/", "status":200, "referer":"", "agent":"MSIE", "userid":3}]}' http://localhost:26578/api/send
# curl -X POST -H "Content-Type: application/json" --data '{"target":"www", "events":[{"path":"/", "status":200, "referer":"", "agent":"Firefox", "userid":4}]}' http://localhost:26578/api/send
# curl -X POST -H "Content-Type: application/json" --data '{"target":"www", "events":[{"path":"/content", "status":200, "referer":"/login", "agent":"MSIE", "userid":3}]}' http://localhost:26578/api/send
# curl -X POST -H "Content-Type: application/json" --data '{"target":"www", "events":[{"path":"/content", "status":200, "referer":"/login", "agent":"Firefox", "userid":4, "search_param":"news worldwide"}]}' http://localhost:26578/api/send
Result of querie with removing events.
# curl -X GET -H "Content-Type: application/json" --data '{"query_name":"www.search"}' http://localhost:26578/api/see
[[1447046270,{"cnt":1}],[1447046280,{"cnt":0}]]

It seems good!

Top image from https://github.com/norikra/norikra.github.io

2015年11月5日木曜日

Mitigating DDoS Attacks nginx with module

nginx module of ngx_http_limit_conn_module and ngx_http_limit_req_module has resistant to DDoS attack.
This nginx modules will provides regulating the incoming HTTP/S traffic and controlling the traffic as it is proxied to backend servers.
This is native nginx module. so settings is easy and performance is nice. If you want to anti DDoS system easily, maybe this solution is good for you.

Let's Try.

HTTP Connection limit
http://nginx.org/en/docs/http/ngx_http_limit_conn_module.html

Set zone name of shared memory and maximum size of this zone.
http {
    # memory limit for number of connections
    limit_conn_zone $remote_addr zone=connection_limit_per_ip:10m;
Allow 10 connection per an IP address at a time.
server {
    # connection limit
    limit_conn connection_limit_per_ip 30;

Let's Check.

Connection number is 10.
It's no Failed.
# ab -c 10 -n 100000 http://127.0.0.1/
Complete requests:      100000
Failed requests:        0
Connection number is 100.
It has Failed.
# ab -c 100 -n 100000 http://127.0.0.1/
Complete requests:      100000
Failed requests:        11866

HTTP request limit
http://nginx.org/en/docs/http/ngx_http_limit_req_module.html

Allow 100 request per an IP address at a one secound.
http {
    # memory limit for number of requests
    limit_req_zone $remote_addr zone=request_limit_per_ip:10m rate=100r/s;
Sets the maximum burst size of requests.
For example, when request is over the 100r/s, max number of connection will be 10.
If you want to no
server {
    # request limit
    limit_req zone=request_limit_per_ip burst=10;

Let's Check.

Connection number is 10. burst=10.
It has no failed. but it has delay.
Time taken for tests is long.
# ab -c 10 -n 1000 http://127.0.0.1/
Time taken for tests:   10.365 seconds
Complete requests:      1000
Failed requests:        0
# grep delay /var/log/nginx/error.log
2015/11/05 15:06:18 [warn] 8595#0: *999 delaying request, excess: 9.700, by zone "request_limit_per_ip", client: 127.0.0.1, server: localhost, request: "GET / HTTP/1.0", host: "127.0.0.1"
2015/11/05 15:06:18 [warn] 8595#0: *1000 delaying request, excess: 9.700, by zone "request_limit_per_ip", client: 127.0.0.1, server: localhost, request: "GET / HTTP/1.0", host: "127.0.0.1"

Connection number is 10. burst=9.
It has failed and delay.
Time taken for tests is short. because many requests has faild.
# ab -c 10 -n 1000 http://127.0.0.1/
Time taken for tests:   0.231 seconds
Complete requests:      1000
Failed requests:        954
# grep delay /var/log/nginx/error.log
2015/11/05 15:07:06 [warn] 8652#0: *912 delaying request, excess: 9.000, by zone "request_limit_per_ip", client: 127.0.0.1, server: localhost, request: "GET / HTTP/1.0", host: "127.0.0.1"
2015/11/05 15:07:06 [warn] 8652#0: *961 delaying request, excess: 9.000, by zone "request_limit_per_ip", client: 127.0.0.1, server: localhost, request: "GET / HTTP/1.0", host: "127.0.0.1"

Connection number is 10. burst=10 nodelay.
If you want to no delay, you can add nodelay opetion.
# ab -c 10 -n 1000 http://127.0.0.1/
Time taken for tests:   0.204 seconds
Complete requests:      1000
Failed requests:        969
# grep delay /var/log/nginx/error.log
Nothing delay.

Exsample Settings of nginx conf
http {
    # Amazon ELB (VPC) 
    set_real_ip_from 172.16.0.0;
    real_ip_header     X-Forwarded-For;

    # memory limit for number of connections
    limit_conn_zone $remote_addr zone=connection_limit_per_ip:10m;

    # memory limit for number of requests
    limit_req_zone $remote_addr zone=request_limit_per_ip:10m rate=1000r/s;

server {
    # connection limit
    limit_conn connection_limit_per_ip 100;

    # request limit
    limit_req zone=request_limit_per_ip burst=10;


nginx logo from NGINX Newsroom - NGINX

2015年11月4日水曜日

Mitigating DDoS Attacks nginx + ngx_mruby + http-dos-detector

https://github.com/matsumoto-r/http-dos-detector
Detect Huge Number of HTTP Requests on Apache and nginx using mruby code.
http-dos-detector use same Ruby code between Apache(mod_mruby) and nginx(ngx_mruby).
It seems, programmable DDoS firewall by mruby on nginx.
This solution provides regulating the incoming HTTP/S traffic and controlling the traffic as it is proxied to backend servers.
Let's try.

Environment:  Amazon Linux AMI 2015.09.1 (HVM), SSD Volume Type - ami-383c1956
c4.xlarge

Update Ruby
(Just update Ruby for Amazon Linux AMI. Actually, no meaning for ngx_mruby. ngx_mruby uses mruby.)
# yum remove ruby*
# yum install ruby22 ruby22-devel rubygem22 rubygem22-rake aws-amitools-ec2
# ruby -v
ruby 2.2.3p173 (2015-08-18 revision 51636) [x86_64-linux-gnu]

Install ngx_mruby with nginx
Why install nginx by yum version? because yum helps some setup operation easily. for example, mkdir, set logrotate, set start script and more.
Use /opt for FHS (http://www.pathname.com/fhs/pub/fhs-2.3.html#OPTADDONAPPLICATIONSOFTWAREPACKAGES).
# yum install git gcc bison openssl-devel pcre-devel nginx
# cd /opt
# git clone git://github.com/matsumoto-r/ngx_mruby.git && cd ngx_mruby
# NGINX_CONFIG_OPT_ENV='--prefix=/opt/nginx' sh build.sh
# make install
# cp /opt/nginx/sbin/nginx /usr/sbin/nginx
# nginx -V
nginx version: nginx/1.9.6
built by gcc 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC)
configure arguments: --add-module=/opt/ngx_mruby --add-module=/opt/ngx_mruby/dependence/ngx_devel_kit --prefix=/opt/nginx

Running "Hello mruby"
# emacs /etc/nginx/nginx.conf
    server {

        location /hello {
            mruby_content_handler_code '
                Server = nginx
                Server.echo "Hello mruby"
            ';
        }

# service nginx start
# curl 127.0.0.1/hello
Hello mruby

Benchmark nginx and ngx_mruby "Hello mruby"
nginx returns static file.
# echo "Hello nginx" > /usr/share/nginx/html/index.html
# ab -c 100 -n 1000000 127.0.0.1/
Requests per second:    38185.31 [#/sec] (mean)
Time per request:       2.619 [ms] (mean)
nginx + ngx_mruby returns "Hello mruby"
# ab -c 100 -n 1000000 127.0.0.1/hello
Requests per second:    36466.16 [#/sec] (mean)
Time per request:       2.742 [ms] (mean)
This is 95.04% power than "nginx static file".

Setup http-dos-detector
Git clone source code.
# cd /opt
# git clone git://github.com/matsumoto-r/http-dos-detector.git
# cp -r /opt/http-dos-detector/dos_detector /etc/nginx/conf.d
Customize dos_detector.rb file. This example is IP address base blocking.
# emacs /etc/nginx/conf.d/dos_detector/dos_detector.rb
Server = get_server_class
r = Server::Request.new
c = Server::Connection.new
cache = Userdata.new.shared_cache
global_mutex = Userdata.new.shared_mutex
host = r.hostname

ip_address = c.remote_ip
if r.headers_in["X-Real-IP"]
  ip_address = r.headers_in["X-Real-IP"]
elsif r.headers_in["X-Forwarded-For"]
  ip_address = r.headers_in["X-Forwarded-For"].split(",").first
end

config = {
  # dos counter by key
  :counter_key => ip_address,
  :magic_str => "....",
  # judging dos access when the number of counter is between :behind_counter and 0
  :behind_counter => -20000,

  # set behind counter when the number of counter is over :threshold_counter
  # in :threshold_time sec
  :threshold_counter => 10000,
  :threshold_time => 5,

  # expire dos counter and initialize counter even
  # if the number of counter is between :behind_counter and 0
  :expire_time => 10,
}

unless r.sub_request?
  # process-shared lock
  timeout = global_mutex.try_lock_loop(50000) do
    dos = DosDetector.new r, cache, config
    data = dos.analyze
    Server.errlogger Server::LOG_NOTICE, "[INFO] dos_detetor: detect dos: #{data}"
    begin
      if dos.detect?
        Server.errlogger Server::LOG_NOTICE, "dos_detetor: detect dos: #{data}"
        Server.return Server::HTTP_SERVICE_UNAVAILABLE
      end
    rescue => e
      raise "DosDetector failed: #{e}"
    ensure
      global_mutex.unlock
    end
  end
  if timeout
    Server.errlogger Server::LOG_NOTICE, "dos_detetor: get timeout mutex lock, #{data}"
  end
end
Modify nginx.conf. add error.log to notice and mruby_init, mruby_init_worker and mruby_access_handler.
# emacs /etc/nginx/nginx.conf

error_log  /var/log/nginx/error.log  notice;

http {

    mruby_init /etc/nginx/conf.d/dos_detector/dos_detector_init.rb cache;
    mruby_init_worker /etc/nginx/conf.d/dos_detector/dos_detector_worker_init.rb cache;

    server {
        location /dos_detector {
            mruby_access_handler /etc/nginx/conf.d/dos_detector/dos_detector.rb cache;
        }
    }
}
# service nginx restart

Benchmark http-dos-detector
nginx + ngx_mruby + dos_detector returns "Hello dos_detector"
# mkdir /usr/share/nginx/html/dos_detector/
# echo "Hello dos_detector" > /usr/share/nginx/html/dos_detector/index.html
Run Apache Bench
# ab -c 100 -n 1000000 127.0.0.1/dos_detector/

Complete requests:      1000000
Non-2xx responses:      800000

Requests per second:    9280.54 [#/sec] (mean)
Time per request:       10.775 [ms] (mean)
"nginx static file" can run much faster this. But, If backend application provides small power than this response (you have no C10k problem), nginx + ngx_mruby + http-dos-detector will be good solution for Mitigating DDoS Attacks.

See also.
ngx_mruby
https://github.com/matsumoto-r/ngx_mruby
mruby-logo provided by h2so5

2015年10月30日金曜日

nginxでDDoS対策をする方法

nginxでDDoS対策をするための方法を調べてみました。
いくつか方法が見つかったので、リンクをまとめてみました。あとで、色々試してみようと思います。
nginxをプロキシサーバとして利用して、バックエンドにアプリサーバを配置するような構成で使うので、外部から来たDDoSをnginxでブロックしてアプリサーバを守る。といったような用途になるかと思います。
もちろん、nginxが応答できるHTTPリクエストの範疇に収まらない攻撃は、その前段階でブロックしたり、他のサーバにアクセスを逃したりする必要があります。
DDoS攻撃に対するシンプルな戦略 | ツチノコブログ(http://tsuchinoko.dmmlabs.com/?p=611)
さらに、こちらで紹介されているように、完璧に防ごうとすると青天井のコストが見えてしまったりするので、”どれくらいDDoSからの保護をするのか”を明確にルールを決めておくと良さそうですね。

以下、リンク集です。

nginx + mruby + http-dos-detector
http-dos-detector
https://github.com/matsumoto-r/http-dos-detector
nginxでもapacheでも使用可能なDoS的アクセスを検知して任意の制御をするWebサーバ拡張をmrubyで作った - 人間とウェブの未来
http://hb.matsumoto-r.jp/entry/2015/06/08/224827
ngx_mrubyでのdos_detector(http-dos-detector)をためし、パフォーマンスに与える影響を測った - Qiita 
http://qiita.com/sawanoboly/items/74368e002631bed3afb7
Mitigating DDoS Attacks nginx + ngx_mruby + http-dos-detector
http://takeshiyako.blogspot.jp/2015/11/mitigating-ddos-attacks-nginx-ngxmruby.html

nginx + mruby + mruby-cgroup
mruby-cgroup
https://github.com/matsumoto-r/mruby-cgroup
RubyKaigi 2014 mod_mruby ngx_mruby // Speaker Deck
https://speakerdeck.com/matsumoto_r/rubykaigi-2014-mod-mruby-ngx-mruby
人間とウェブの未来 - mruby-cgroupとmod_mrubyでApacheのリソースを制御、そこから得られるcgroupの挙動とは
http://blog.matsumoto-r.jp/?p=3285
mod_mruby で特定のCGIのCPU使用率を制限してみたら感動した話 - フリーエンジニアライフ
http://takeyuweb.hatenablog.com/entry/2014/11/22/120335

nginx + nginx-maxconn-module
nginx-maxconn-module
https://github.com/cybozu/nginx-maxconn-module
nginx の拡張モジュールを書いて DoS 対策をした - Cybozu Inside Out | サイボウズエンジニアのブログ
http://blog.cybozu.io/entry/8363

nginx conf
nginx - ELB内のngixでDDoS対策 - Qiita
http://qiita.com/drobune/items/fce3e3629a472dc3237e
Nginx Tuning For Best Performance
https://gist.github.com/denji/8359866#nginx-simple-ddos-defense
Nginxで使える制限一覧 | しのぶら跡地
http://shinobra.com/2012/03/sawanoboly/nginx_limiter
コーポレートサイトのNginxの設定をチューニングも兼ねていじった - onigra.github.io
http://onigra.github.io/blog/2015/01/07/corporate-site-tuning-with-nginx/
Mitigating DDoS Attacks nginx with module
http://takeshiyako.blogspot.jp/2015/11/mitigating-ddos-attacks-nginx-module.html


AWSを使っているならば下記のスライドが参考になります。
nginxのログからブラックリストを作ってアクセスコントロールに追加したりとかは、お手軽にできそうですね。
DDoS Resiliency with Amazon Web Services (SEC305) | AWS re:Invent 2013
http://www.slideshare.net/AmazonWebServices/ddos-resiliency-with-amazon-web-services-sec305-aws-reinvent-2013
(SEC307) Building a DDoS-Resilient Architecture with Amazon Web Services | AWS re:Invent 2014
http://www.slideshare.net/AmazonWebServices/sec307-building-a-ddosresilient-architecture-with-amazon-web-services-aws-reinvent-2014
AWSでブラックリスト型のFirewallを実装する。〜BlackList Firewallパターン〜 - プログラマになりたい
http://blog.takuros.net/entry/20131213/1386944370



Top image from BlinkenArea.org 28c3-ddos-use.more.bandwith

2015年10月28日水曜日

PHP with Amazon DynamoDB

This blog entry aim to construct the good connectivity performance from PHP to Amazon DynamoDB.

Make DynamoDB table
Table Name: ProductCatalog
Primary Hash Key: Id (String)
Provisioned Read Capacity Units: 10000
Provisioned Write Capacity Units: 10000
Region: Asia Pacific (Tokyo)

Launch EC2 Instance
AMI ID: Amazon Linux AMI 2015.09 (HVM), SSD Volume Type - ami-9a2fb89a
Instance type: c4.xlarge

Install PHP and AWS SDK for PHP
Install PHP5.6
# yum -y install php56 php56-devel php56-fpm php56-opcache gcc
Make app dir and install new version of aws-sdk-php by Composer.
# mkdir ~/sample_app; cd ~/sample_app
# curl -sS https://getcomposer.org/installer | php
# php composer.phar require aws/aws-sdk-php
# php composer.phar require doctrine/cache
# php composer.phar install --optimize-autoloader
# yum -y install php-pear
# pecl install uri_template
# echo "extension=uri_template.so" >> /etc/php.ini
# php composer.phar show --installed | grep aws/aws-sdk-php
aws/aws-sdk-php        3.9.1 AWS SDK for PHP - Use Amazon Web Services in your PHP project
Make AWS credentials file.
# mkdir ~/.aws/
# emacs ~/.aws/credentials
[default]
aws_access_key_id = YOUR_AWS_ACCESS_KEY_ID
aws_secret_access_key = YOUR_AWS_SECRET_ACCESS_KEY
Make sample put app
# emacs put.php
<?php
require 'vendor/autoload.php';
use Aws\DynamoDb\DynamoDbClient;
$client = DynamoDbClient::factory(array(
    'profile' => 'default',
    'region' => 'ap-northeast-1',
    'version' => '2012-08-10',
    'credentials.cache' => true,
    'validation' => false,
    'scheme' => 'http'
));
$response = $client->putItem(array(
    'TableName' => 'ProductCatalog',
    'Item' => array(
        'Id'       => array('S'      => '104'      ), // Primary Key
        'Title'    => array('S'      => 'Book 104 Title' ),
        'ISBN'     => array('S'      => '111-1111111111' ),
        'Price'    => array('N'      => '25' ),
        'Authors'  => array('SS'  => array('Author1', 'Author2') )
    )
));
Run.
# php put.php

Make sample get app
# emacs get.php
<?php
require 'vendor/autoload.php';
use Aws\DynamoDb\DynamoDbClient;
$client = DynamoDbClient::factory(array(
    'profile' => 'default',
    'region' => 'ap-northeast-1',
    'version' => '2012-08-10',
    'credentials.cache' => true,
    'validation' => false,
    'scheme' => 'http'
));
$response = $client->getItem(array(
    'TableName' => 'ProductCatalog',
    'Key' => array(
 'Id' => array( 'S' => '104' )
    )
));
print_r ($response['Item']);
Run
# php get.php
Array
(
    [ISBN] => Array
        (
            [S] => 111-1111111111
        )

    [Id] => Array
        (
            [S] => 104
        )

    [Authors] => Array
        (
            [SS] => Array
                (
                    [0] => Author1
                    [1] => Author2
                )

        )

    [Price] => Array
        (
            [N] => 25
        )

    [Title] => Array
        (
            [S] => Book 104 Title
        )

)


Benchmark
Setup Nginx.
# yum -y install nginx
# mkdir /var/lib/nginx/.aws/
# cp ~/.aws/credentials /var/lib/nginx/.aws/credentials
# cp -r ~/sample_app/* /usr/share/nginx/html/
# emacs /etc/php-fpm.d/www.conf
user = apache
group = apache
->
user = nginx
group = nginx

# emacs /etc/nginx/nginx.conf
    server {
        include /etc/nginx/default.d/*.conf;
 
# /etc/init.d/nginx start
# /etc/init.d/php-fpm start
# curl 127.0.0.1/get.php -v
Run benchmark.
# ab -c 100 -n 10000 http://127.0.0.1/put.php
Requests per second:    763.44 [#/sec] (mean)

# ab -c 100 -n 10000 http://127.0.0.1/get.php
Requests per second:    832.62 [#/sec] (mean)
Maybe. If I install AWS SDK with ClassPreloader, that will be more faster.
But, ClassPreloader does not work with AWS SDK for PHP version 3.
Please let me know, if you know how to use ClassPreloader.
Next section. I tried a other way.

Benchmark with Go DynamoDB Proxy
Use bbpd.
http://takeshiyako.blogspot.jp/2015/10/go-lang-bbpd-godynamo-http-proxy.html

Make sample apps.
# emacs /usr/share/nginx/html/get_go.php
<?php
$url = "http://localhost:12333/GetItemJSON";
$postdata = "{\"Key\": {\"Id\": {\"S\": \"104\"}}, \"TableName\": \"ProductCatalog\"}";
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $postdata);
$result = curl_exec($ch);
curl_close($ch);
echo $result;

# emacs /usr/share/nginx/html/put_go.php
<?php
$random = substr(str_shuffle('1234567890abcdefghijklmnopqrstuvwxyz'), 0, 36);
$url = "http://localhost:12333/PutItemJSON";
$postdata = "{\"Item\": {\"Id\": \"$random\",\"Title\":\"Book 107 Title\",\"ISBN\":\"111-1111111111\",\"Price\":\"33\",\"Authors\":[\"Author1\", \"Author2\"]},\"TableName\": \"ProductCatalog\"}";
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $postdata);
$result = curl_exec($ch);
curl_close($ch);
echo $result;
Run benchmark.
# ab -c 100 -n 10000 http://127.0.0.1/get_go.php
Requests per second:    4260.64 [#/sec] (mean)

# ab -c 100 -n 10000 http://127.0.0.1/put_go.php
Requests per second:    3753.05 [#/sec] (mean)
It seems this solution is better than AWS SDK for PHP.
NOTE: When update same Id, many RETRY occurs with authreq. result of benchmark is 750 req/s.
if you update many data, please carefully.
https://github.com/smugmug/godynamo/blob/master/authreq/authreq.go#L91

See also
AWS SDK for PHP 3.x
http://docs.aws.amazon.com/aws-sdk-php/v3/api/index.html
Requirements — AWS SDK for PHP documentation
http://docs.aws.amazon.com/aws-sdk-php/v3/guide/getting-started/requirements.html
Performance Guide — AWS SDK for PHP 2.8.22 documentation
http://docs.aws.amazon.com/aws-sdk-php/v2/guide/performance.html
Top image from Felix De Vliegher Enjoying the view

2015年10月27日火曜日

Go lang hot-deploy with go-server-starter

https://github.com/lestrrat/go-server-starter
The start_server utility is a superdaemon for hot-deploying server programs.
Let's try.
Environment: EC2 AMI ID: Amazon Linux AMI 2015.09 (HVM), SSD Volume Type - ami-9a2fb89a

Go Sample app
Install Go
$ sudo yum -y install golang
$ go version
go version go1.4.2 linux/amd64
$ mkdir ~/gocode
$ echo 'export GOPATH=$HOME/gocode' >> ~/.bashrc
$ source $HOME/.bashrc
Install go-server-starter
# go get github.com/lestrrat/go-server-starter/cmd/start_server
# $GOPATH/bin/start_server --version
0.0.2
# $GOPATH/bin/start_server --help
Make sample web app
# mkdir $GOPATH/src/web/
# cd $GOPATH/src/web/
# emacs sample.go
package main

import (
 "fmt"
 "github.com/lestrrat/go-server-starter/listener"
 "log"
 "net"
 "net/http"
 "os"
)

func hello(w http.ResponseWriter, r *http.Request) {
 fmt.Fprintf(w, "Hello world!")
}

func newHandler() http.Handler {
 mux := http.NewServeMux()
 mux.HandleFunc("/", hello)
 return mux
}

func main() {
 var l net.Listener

 // check environment variable
 if os.Getenv("SERVER_STARTER_PORT") != "" {
  listeners, err := listener.ListenAll()
  if err != nil {
   fmt.Println(err)
   return
  }

  if 0 < len(listeners) {
   l = listeners[0]
  }
 }

 // default port
 if l == nil {
  var err error
  l, err = net.Listen("tcp", fmt.Sprintf(":8080"))

  if err != nil {
   fmt.Println(err)
   return
  }
 }

 log.Printf("Start Go HTTP Server")
 fmt.Println(http.Serve(l, newHandler()))

}

# go install
# $GOPATH/bin/web
2015/10/26 07:36:23 Start Go HTTP Server
# curl localhost:8080
Hello world!

Start app with start_server
# KILL_OLD_DELAY=5 nohup $GOPATH/bin/start_server --port=8000 --pid-file=$GOPATH/start_server.pid -- $GOPATH/bin/web >> $GOPATH/start_server.log &

# cat $GOPATH/start_server.log
starting new worker 26499
2015/10/26 08:36:55 Start Go HTTP Server

$ curl http://127.0.0.1:8000/
Hello world!

Try to hot-deploy.
# cd $GOPATH/src/web/
# emacs sample.go
        fmt.Fprintf(w, "Hello world!")
->
        fmt.Fprintf(w, "Hello world!!!!")
# go install
Graceful restart.
# /bin/cat $GOPATH/start_server.pid | xargs kill -HUP
Check log.
# cat $GOPATH/start_server.log
received HUP (num_old_workers=TODO)
spawning a new worker (num_old_workers=TODO)
starting new worker 2618
2015/10/27 02:43:21 Start Go HTTP Server
new worker is now running, sending TERM to old workers:2576
sleep 5 secs
killing old workers
old worker 2576 died, status:512
Check result.
$ curl http://127.0.0.1:8000/
Hello world!!!!
That's fine.

No error with graceful restart?
Set Nginx for Reverse Proxy
Nginx is necessary for safe connection from the HTTP cliant. Nginx Reverse Proxy makes re-handshake for the backend.
$ sudo yum -y install nginx
$ sudo emacs /etc/nginx/nginx.conf
worker_processes  1;
events {
    worker_connections  50000;
}
http {
    include     mime.types;
    access_log  off;
    upstream app {
        server 127.0.0.1:8000 max_fails=10;
        server localhost:8000 max_fails=10 backup;
    }
    server {
        location / {
            proxy_pass http://app;
            proxy_next_upstream error http_502;
        }
    }
}
$ sudo service nginx start
$ curl http://127.0.0.1/
Hello world!!!!
Run long time ab
$ sudo yum -y install httpd-tools
# ab -c 10 -n 200000 http://127.0.0.1/
Graceful restart
$ /bin/cat $GOPATH/start_server.pid | xargs kill -HUP
$ tail -f  $GOPATH/start_server.log
received HUP (num_old_workers=TODO)
spawning a new worker (num_old_workers=TODO)
starting new worker 2766
2015/10/27 02:56:48 Start Go HTTP Server
new worker is now running, sending TERM to old workers:2759
sleep 5 secs
killing old workers
old worker 2759 died, status:512
Result from ab
Failed requests:        0

Benchmark
start_server support UNIX domain socket. Let's Benchmark Port vs UNIX domain socket.
Instance type: c4.xlarge
Benchmark command.
# ab -c 100 -n 1000000 http://127.0.0.1/
start_server start commands:
Port
# KILL_OLD_DELAY=5 nohup $GOPATH/bin/start_server --port=8000 --pid-file=$GOPATH/start_server.pid -- $GOPATH/bin/web >> $GOPATH/start_server.log &

UNIX domain socket
$ KILL_OLD_DELAY=5 nohup $GOPATH/bin/start_server --path /tmp/app.sock --pid-file=$GOPATH/start_server.pid -- $GOPATH/bin/web >> $GOPATH/start_server.log &
$ chmod 777 /tmp/app.sock
Nginx confs:
Port
    upstream app {
        server 127.0.0.1:8000;
    }

UNIX domain socket
    upstream app {
        server unix:/tmp/app.sock;
    }
Results.
Port
Requests per second:    12544.89 [#/sec] (mean)

UNIX domain socket
Requests per second:    18435.59 [#/sec] (mean)

Tips
How to stop start_server
# /bin/cat $GOPATH/start_server.pid | xargs kill -TERM
# cat $GOPATH/start_server.log
received TERM, sending TERM to all workers:2766
worker 2766 died, status:512
exiting

See also.
Server Starter - a superdaemon to hot-deploy server programs
http://www.slideshare.net/kazuho/server-starter-a-superdaemon-to-hotdeploy-server-programs
Fix panic when SERVER_STARTER_PORT is not defined by handlename · Pull Request #3 · lestrrat/go-server-starter
https://github.com/lestrrat/go-server-starter/pull/3
Docker と SO_REUSEPORT を組み合わせてコンテナのHot Deployにチャレンジ - blog.nomadscafe.jp
http://blog.nomadscafe.jp/2015/01/docker-and-so-reuseport-hot-deploy.html
Server::Starterから学ぶhot deployの仕組み - $shibayu36->blog;
http://blog.shibayu36.org/entry/2012/05/07/201556
Top image from John Lord Starter

2015年10月23日金曜日

Go lang bbpd is GoDynamo HTTP Proxy

https://github.com/smugmug/bbpd
bbpd is a http proxy server for Amazon's DynamoDB service.
bbpd uses GoDynamo package for the Amazon DynamoDB database. It made by smugmug.
It supports JSON documents and can post/get data through from HTTP POST method. Oh, that is pretty nice.
So let's try to make DynamoDB HTTP Proxy.

Environment: EC2 AMI ID: Amazon Linux AMI 2015.09 (HVM), SSD Volume Type - ami-9a2fb89a

Install Go
$ sudo yum -y install golang zsh
$ go version
go version go1.4.2 linux/amd64
$ mkdir ~/gocode
$ echo 'export GOPATH=$HOME/gocode' >> ~/.bashrc
$ source $HOME/.bashrc

Install bbpd
$ go get github.com/smugmug/bbpd
$ sudo cp $GOPATH/bin/bbpd /usr/bin/
$ sudo cp $GOPATH/src/github.com/smugmug/bbpd/bin/bbpd/bbpd_ctl /usr/bin/
$ sudo cp $GOPATH/src/github.com/smugmug/bbpd/bin/bbpd/bbpd_daemon /usr/bin/
$ sudo chmod 755 /usr/bin/bbpd_ctl
$ sudo chmod 755 /usr/bin/bbpd_daemon
Set config file.
$ cp $GOPATH/src/github.com/smugmug/godynamo/conf_file/test_aws-config.json ~/.aws-config.json
$ emacs ~/.aws-config.json
{
    "extends":[],
    "services": {
        "default_settings":{
            "params":{
                "access_key_id":"myAccessKey",
                "secret_access_key":"mySecret",
                "use_sys_log":true
            }
        },
        "dynamo_db": {
            "host":"dynamodb.us-east-1.amazonaws.com",
            "zone":"us-east-1",
            "scheme":"http",
            "port":80,
            "keepalive":false,
            "iam": {
                "use_iam":false,
                "role_provider":"",
                "access_key":"",
                "secret_key":"",
                "token":"",
                "base_dir":"",
                "watch":false
            }
        }
    }
}
Start.
$ bbpd_ctl start
**** starting bbpd as daemon
bbpd - started
2015/10/23 02:22:17 global conf.Vals initialized
2015/10/23 02:22:17 not using iam, assume credentials hardcoded in conf file
2015/10/23 02:22:17 starting bbpd...
2015/10/23 02:22:17 induce panic with ctrl-c (kill -2 2801) or graceful termination with kill -[1,3,15] 2801
2015/10/23 02:22:17 trying to bind to port:12333
2015/10/23 02:22:17 init routing on port 12333
2015/10/23 02:22:17 global conf.Vals initialized
2015/10/23 02:22:17 not using iam, assume credentials hardcoded in conf file
2015/10/23 02:22:17 starting bbpd...
2015/10/23 02:22:17 induce panic with ctrl-c (kill -2 2804) or graceful termination with kill -[1,3,15] 2804
2015/10/23 02:22:17 trying to bind to port:12333
2015/10/23 02:22:17 port 12333 already in use
2015/10/23 02:22:17 trying to bind to port:12334
2015/10/23 02:22:17 init routing on port 12334
Check processes.
$ pgrep -l bbpd
24330 bbpd
24333 bbpd
Get status.
$ curl -H "X-Bbpd-Indent: true" "http://localhost:12333/Status"
{
 "Status": "ready",
 "AvailableHandlers": [
  "/DescribeTable/",
  "/DeleteItem",
  "/ListTables",
  "/CreateTable",
  "/UpdateTable",
  "/StatusTable/",
  "/PutItem",
  "/PutItemJSON",
  "/GetItem",
  "/GetItemJSON",
  "/BatchGetItem",
  "/BatchGetItemJSON",
  "/BatchWriteItem",
  "/BatchWriteItemJSON",
  "/UpdateItem",
  "/Query",
  "/Scan",
  "/RawPost/",
  "/"
 ],
 "Args": {
  "X-Bbpd-Indent": "set '-H \"X-Bbpd-Indent: True\" ' to indent the top-level json",
  "X-Bbpd-Verbose": "set '-H \"X-Bbpd-Verbose: True\" ' to get verbose output"
 },
 "Summary": {
  "StartTime": "2015-10-23 08:12:18.875147234 +0000 UTC",
  "RunningTime": "11.531750029s",
  "LongestResponse": "0.00ms",
  "AverageResponse": "0.00ms",
  "LastResponse": "no requests made yet",
  "ResponseCount": "0"
 }
}
Now, bbpd is ready.

Run curl
Sample scripts are here. https://github.com/smugmug/bbpd/tree/master/tests

Create table
$ curl -H "X-Amz-Target: DynamoDB_20120810.CreateTable" -X POST -d '
{
    "AttributeDefinitions": [
        {
            "AttributeName": "user_id", 
            "AttributeType": "S"
        }
    ], 
    "KeySchema": [
        {
            "AttributeName": "user_id", 
            "KeyType": "HASH"
        }
    ], 
    "ProvisionedThroughput": {
        "ReadCapacityUnits": 100, 
        "WriteCapacityUnits": 100
    }, 
    "TableName": "bbpd-test"
}
' http://localhost:12333/
Put
curl -H "X-Amz-Target: DynamoDB_20120810.PutItem"  -X POST -d '
{
    "Item": {
        "num": 1, 
        "numlist": [
            6, 
            7, 
            1, 
            2, 
            3, 
            9, 
            -7234234234.234234
        ], 
        "stringlist": [
            "pk1_a", 
            "pk1_b", 
            "pk1_c"
        ], 
        "user_id": "my_user_id1"
    }, 
    "TableName": "bbpd-test"
}
' http://localhost:12333/PutItemJSON
Get
$ curl -H "X-Amz-Target: DynamoDB_20120810.GetItem" -X POST -d '
{
    "Key": {
        "user_id": {
            "S": "my_user_id1"
        }
    }, 
    "TableName": "bbpd-test"
}
' "http://localhost:12333/GetItemJSON"

{"Item":{"num":1,"numlist":[-7.234234234234234e+09,3,9,7,2,1],"stringlist":["pk1_c","pk1_a","pk1_b"],"user_id":"my_user_id1"}}
Looks fine!

Check table by SQL using ddbcli
https://github.com/winebarrel/ddbcli
$ gem install ddbcli --no-ri --no-rdoc
Fetching: ddbcli-0.5.3.gem (100%)
$ ddbcli -k ACCESS_KEY -s SECRET_KEY -r REGION_OR_ENDPOINT
ap-northeast-1> select all * from bbpd-test;
[
  {"num":1,"numlist":[6,3,9,7,2,1,-7234234234.234234],"stringlist":["pk1_a","pk1_b","pk1_c"],"user_id":"my_user_id1"}
]
// 1 row in set (0.02 sec)
That's correct.

Benchmark
DynamoDB
Provisioned Read Capacity Units: 10000
Provisioned Write Capacity Units: 10000

EC2
Instance type: c4.large, c4.xlarge, c4.2xlarge
# emacs postdata.file
{
    "Key": {
        "user_id": {
            "S": "my_user_id1"
        }
    },
    "TableName": "bbpd-test"
}
# ab -n 100000 -c 100 -p postdata.file -T "application/json" http://localhost:12333/GetItemJSON

c4.large
Requests per second:    4754.25 [#/sec] (mean)

c4.xlarge
Requests per second:    8591.35 [#/sec] (mean)

c4.2xlarge
Requests per second:    8540.63 [#/sec] (mean) 
Many RETRY occurs with godynamo authreq.retryReq.

Tips
Set log dir.
$ sudo emacs /usr/bin/bbpd_ctl

LOG=/path/to/log/bbpd.log

    start)
        echo -n "**** starting bbpd as daemon\n"
        $INSTALL_PATH/$DAEMON >> $LOG 2>&1 || true
        $INSTALL_PATH/$DAEMON >> $LOG 2>&1 || true
        echo "bbpd - started\n"
        ;;

Amazon DynamoDB logo / Amazon Web Services LLC - http://aws.typepad.com/aws/2011/12/introducing-aws-simple-icons-for-your-architecture-diagrams.html
gopher-side_color.{ai,svg,png} was created by Takuya Ueda (http://u.hinoichi.net). Licensed under the Creative Commons 3.0 Attributions license. - https://github.com/golang-samples/gopher-vector#gopher-side_color
The Go gopher was designed by Renee French. (http://reneefrench.blogspot.com/)

2015年10月22日木曜日

Go lang Goji + Einhorn provides "graceful restart"


Goji aka WAF of Golang.
Says.
Automatic support for Einhorn, systemd, and more
Graceful shutdown, and zero-downtime graceful reload when combined with Einhorn.
Looks very pritty.
So, Let's try Goji, and graceful restart by Einhorn.

Make Goji app.
Get Goji.
$ go get github.com/zenazn/goji
Make sample goji app.
$ mkdir ~/gocode/src/goji_sample/
$ cd ~/gocode/src/goji_sample/
$ emacs sample1.go
package main

import (
 "fmt"
 "net/http"

 "github.com/zenazn/goji"
 "github.com/zenazn/goji/web"
)

func hello(c web.C, w http.ResponseWriter, r *http.Request) {
 fmt.Fprintf(w, "Hello?, %s!", c.URLParams["name"])
}

func main() {
 goji.Get("/hello/:name", hello)
 goji.Serve()
}
$ go install

Set up Einhorn and run app.
Install Einhorn with gem.
$ sudo gem install einhorn
Start Goji app with Einhorn.
$ nohup einhorn -b 0.0.0.0:8001 -c einhorn_with_goji_sample -m manual $GOPATH/bin/goji_sample >> /tmp/app.log&

It works?
Chak HTTP result via curl.
$ curl 0.0.0.0:8001/hello/aaa
Hello?, aaa!
It warks!
Check app log.
$ tail -f /tmp/app.log
[MASTER 26118] INFO: Writing PID to /var/folders/6_/n62p97qx08x53ncc8hqn5fvm0000gp/T/einhorn-einhorn_with_sample_goji.pid
[MASTER 26118] INFO: Binding to 0.0.0.0:8001 with flags []
[MASTER 26118] INFO: Launching 1 new workers
[MASTER 26118] INFO: ===> Launched 26122 (index: 0)
[WORKER 26122] INFO: About to exec ["/Users/yako/gocode/bin/goji_sample"]
2015/10/21 18:11:35.258055 Starting Goji on 0.0.0.0:8001
2015/10/21 18:11:35.258153 bind: ACKing to einhorn
[MASTER 26118] INFO: [client 2:11] client connected
[MASTER 26118] INFO: Received a manual ACK from 26122
[MASTER 26118] INFO: Up to 1 / 1 manual ACKs
[MASTER 26118] INFO: [client 2:11] client disconnected
2015/10/21 18:11:49.985358 [ore-no-MacBook-2.local/zVchqfLPzY-000001] Started GET "/hello/aaa" from 127.0.0.1:61537
2015/10/21 18:11:49.986692 [ore-no-MacBook-2.local/zVchqfLPzY-000001] Returning 200 in 367.506µs
Check process.
$ ps | grep goji
26197 ttys000    0:00.22 einhorn: ruby einhorn_with_goji_sample
26201 ttys000    0:00.01 /Users/yako/gocode/bin/goji_sample

Graceful restart Goji with Einhorn
Modify current sample app.
$ emacs sample1.go
        fmt.Fprintf(w, "Hello?, %s!", c.URLParams["name"])
->
        fmt.Fprintf(w, "Hello!!!, %s!", c.URLParams["name"])
$ go install
Run Einhorn upgrade command. this command do graceful restart Goji app.
$ einhornsh -c einhorn_with_goji_sample -e upgrade
Welcome, yako! You are speaking to Einhorn Master Process 26197 (einhorn_with_goji_sample).
This is Einhorn 0.6.4.
Upgrading smoothly, as commanded
Starting smooth upgrade from version 0...
===> Launched 26273 (index: 1)
Upgraded successfully to version 1 (Einhorn 0.6.4).
Upgrade done
$ curl 0.0.0.0:8001/hello/aaa
Hello!!!, aaa!

No error with graceful restart?
Run long time ab(Apache Bench) and then run graceful result.
$ ab -c 1 -n 10000 http://0.0.0.0:8001/hello/aaa
$ einhornsh -c einhorn_with_goji_sample -e upgrade
Chcek result of ab.
Failed requests:        0
looks fine.

Tips
How to shotdown Einhorn worker.
$  einhornsh -c einhorn_with_goji_sample -e die
Welcome, yako! You are speaking to Einhorn Master Process 26529 (einhorn_with_goji_sample).
This is Einhorn 0.6.4.
Einhorn is going down! Successfully sent USR2s to 1 processes: [26586]
$ tail -f /tmp/app.log
[MASTER 26529] INFO: [client 4:11] client connected
[MASTER 26529] INFO: Sending USR2 to [26586]
2015/10/21 19:24:57.165737 Goji received signal, gracefully stopping
2015/10/21 19:24:57.165790 Goji stopped
[MASTER 26529] INFO: ===> Exited worker 26586
[MASTER 26529] INFO: [client 4:11] client disconnected

See also.
Build and Deploy to production · Issue #105 · zenazn/goji
https://github.com/zenazn/goji/issues/105
Meet Einhorn
https://stripe.com/blog/meet-einhorn
lestrrat/go-server-starter Go port of start_server utility (Server::Starter)
https://github.com/lestrrat/go-server-starter
fvbock/endless Zero downtime restarts for golang HTTP and HTTPS servers.
https://github.com/fvbock/endless
guregu/kami web "framework" using x/net/context
https://github.com/guregu/kami
facebookgo/grace Graceful restart for Go servers.
https://github.com/facebookgo/grace
dcu/http-einhorn
https://github.com/dcu/http-einhorn
Top image form Panagiotis Giannakopoulos

2015年10月21日水曜日

Go lang Syntax highlighting with Emacs Mac OS X

This example provides Syntax highlighting support for Google Go Language (golang) in Emacs.
I use El-Get with Emacs.

Syntax highlighting
Add El-Get code to Emacs int.el file.
$ emacs ~/.emacs.d/init.el
(when load-file-name
  (setq user-emacs-directory (file-name-directory load-file-name)))

(add-to-list 'load-path (locate-user-emacs-file "el-get/el-get"))
(unless (require 'el-get nil 'noerror)
  (with-current-buffer
      (url-retrieve-synchronously
       "https://raw.githubusercontent.com/dimitri/el-get/master/el-get-install.el")
    (goto-char (point-max))
    (eval-print-last-sexp)))
    ;; golang
    (el-get-bundle go-mode)
Open .go file.
$ emacs sample.go
Thats it.

goimports
https://godoc.org/golang.org/x/tools/cmd/goimports
Command goimports updates your Go import lines, adding missing ones and removing unreferenced ones.

Lets set goimports..
Add GOPATH and PATH to .bashrc. GOPATH is ~/gocode.
$ emacs ~/.bashrc
export GOPATH=$HOME/gocode
 if [ -d $GOPATH/bin ]; then
     PATH=$PATH:$GOPATH/bin
 fi
$ source $HOME/.bashrc
Get goimports.
$ go get golang.org/x/tools/cmd/goimports
Set ~/.emacs.d/init.el.
$ emacs ~/.emacs.d/init.el
(when load-file-name
  (setq user-emacs-directory (file-name-directory load-file-name)))

(add-to-list 'load-path (locate-user-emacs-file "el-get/el-get"))
(unless (require 'el-get nil 'noerror)
  (with-current-buffer
      (url-retrieve-synchronously
       "https://raw.githubusercontent.com/dimitri/el-get/master/el-get-install.el")
    (goto-char (point-max))
    (eval-print-last-sexp)))
    (el-get-bundle go-mode)

(add-hook 'before-save-hook 'gofmt-before-save)
(let ((goimports (executable-find "goimports")))
  (if goimports (setq gofmt-command goimports)))
Lets make new sample code. this code has no import and no Indentation for exsample.
$ mkdir $GOPATH/src/sample
$ emacs $GOPATH/src/sample/sample_code.go
package main

func main() {
fmt.Printf(stringutil.Reverse("!oG ,olleH"))
}
Run Emacs save command.
C-x C-s
package main

import "fmt"

func main() {
        fmt.Printf(stringutil.Reverse("!oG ,olleH"))
}
The new line of import was added automaticaly. Nice!

2015年10月20日火曜日

Go langでrandを使って擬似乱数を作る

Go langで擬似乱数を作ってみます。
簡単なコードとしては以下のようになります。
0から9までの数字を10回出力します。
package main

import (
    "fmt"
    "math/rand"
    "time"
)

func main() {
    rand.Seed(time.Now().UnixNano())
    for i := 0; i < 10; i++ {
        fmt.Println("My favorite number is", rand.Intn(10))
    }
}
$ go run rand.go
My favorite number is 0
My favorite number is 7
My favorite number is 0
My favorite number is 8
My favorite number is 3
My favorite number is 3
My favorite number is 7
My favorite number is 5
My favorite number is 0
My favorite number is 8
rand.Seed(time.Now().UnixNano()) という箇所で乱数の元になるシードを与えるのがポイントです。
なぜ、シードを与えるかというと、ここで使っている関数rand.Intnは、randパッケージに新しい乱数ジェネレータ生成するときに同じ初期化 rand.NewSource(1) をしているそうで、シードをあたえて、異なる乱数を出す必用があるからだそうです。

以下、シードを与えていないと、同じ結果になる例です。
func main() {
    // シードを与えないで実行
    for i := 0; i < 10; i++ {
        fmt.Print(rand.Intn(10), " ")
    }
    fmt.Println(" ")

    // randをNewSourceで初期化して実行
    my_rand := rand.New(rand.NewSource(1))
    for i := 0; i < 10; i++ {
        fmt.Print(my_rand.Intn(10), " ")
    }
    fmt.Println(" ")

    // シードを与えて実行
    my_rand.Seed(time.Now().UnixNano())
    for i := 0; i < 10; i++ {
        fmt.Print(my_rand.Intn(10), " ")
    }
}
$ go run rand.go
1 7 7 9 1 8 5 0 6 0
1 7 7 9 1 8 5 0 6 0
8 7 1 0 1 4 9 0 3 0
$ go run rand.go
1 7 7 9 1 8 5 0 6 0
1 7 7 9 1 8 5 0 6 0
4 0 4 7 2 0 2 9 7 0

乱数を使って抽選を実装したlotteryという面白いライブラリを作っている方がいたので、遊んでみました。
go get実行を実行してgithubのコードを取得。
$ go get github.com/kyokomi/lottery
$ tree ~/gocode/src/github.com/
/Users/yako/gocode/src/github.com/
└── kyokomi
    └── lottery
        ├── LICENSE
        ├── README.md
        ├── circle.yml
        ├── lottery.go
        └── lottery_test.go

2 directories, 5 files
1/5の確率でアタリ、それ以外をハズレとして、1000回出力。アタリが出た確率も最後に出力します。
package main

import (
        "fmt"
        "github.com/kyokomi/lottery"
        "math/rand"
        "time"
)

func main() {
        lot := lottery.New(rand.New(rand.NewSource(time.Now().UnixNano())))

        check := 1000
        count := 0

        for i := 0; i < check; i++ {

                if lot.LotOf(1, 5) {
                        // 1/5の確率
                        fmt.Print("アタリ")
                        count++
                } else {
                        fmt.Print("ハズレ")
                }

        }

        fmt.Println(float32(count)/float32(check)*100, "%")

}
$ go run lottery.go
ハズレハズレハズレアタリアタリアタリハズレハズレハズレアタリハズレアタリハズレハズレハズレハズレハズレハズレハズレハズレアタリハズレハズレハズレハズレハズレアタリハズレハズレハズレハズレハズレハズレハズレハズレハズレアタリハズレハズレハズレハズレハズレハズレアタリハズレアタリハズレハズレハズレハズレアタリハズレハズレアタリハズレアタリハズレアタリハズレハズレアタリアタリハズレアタリハズレハズレアタリハズレハズレハズレハズレハズレハズレハズレハズレハズレアタリハズレハズレハズレハズレアタリハズレハズレハズレハズレハズレアタリアタリハズレハズレハズレアタリハズレハズレアタリアタリハズレハズレハズレハズレハズレハズレハズレアタリハズレハズレハズレハズレハズレハズレハズレハズレハズレハズレハズレハズレハズレハズレハズレアタリハズレハズレハズレハズレハズレハズレハズレハズレハズレハズレハズレハズレハズレハズレハズレハズレハズレハズレハズレハズレハズレアタリハズレアタリハズレハズレハズレアタリハズレハズレハズレハズレハズレアタリアタリハズレアタリハズレハズレハズレハズレハズレハズレハズレハズレハズレハズレアタリアタリハズレアタリハズレハズレハズレハズレハズレハズレアタリアタリハズレハズレハズレハズレアタリハズレハズレアタリ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
19.9 %
抽選プログラムを作る機会があったら使ってみるとよさそうです。

参考文献)
rand - The Go Programming Language
https://golang.org/pkg/math/rand/
お気楽 Go 言語プログラミング入門 並列プログラミング
Goでlotteryという抽選ライブラリを作った - きょこみのーと
http://kyokomi.hatenablog.com/entry/2015/07/26/151138

Top image from Anthony Starks Gopher inspecting Go code
The Go gopher was designed by Renee French. http://reneefrench.blogspot.com/

2015年10月19日月曜日

Go langのnet/httpでHTTPサーバーを立てる

Go langは、簡単にウェブサーバを立てられるという話を聞いたので、ウェブサーバを立ててみます。
ここでは、Go langのhttpパッケージを使いました。

Hello, World
まずは、簡単に"Hello, World"を返すサーバです。
package main

import (
    "fmt"
    "net/http"
    "log"
)

// http.HandleFuncに登録する関数
// http.ResponseWriterとhttp.Requestを受ける
func HelloServer(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintf(w, "Hello, World\n")
}

func TestServer(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintf(w, "TEST\n")
}

func main() {
    // http.HandleFuncにルーティングと処理する関数を登録
    http.HandleFunc("/", HelloServer) 
    http.HandleFunc("/test", TestServer)

    // ログ出力
    log.Printf("Start Go HTTP Server")

    // http.ListenAndServeで待ち受けるportを指定
    err := http.ListenAndServe(":4000", nil)

    // エラー処理
    if err != nil {
       log.Fatal("ListenAndServe: ", err)
    }
}
go run
サーバスタート
$ go run web.go
2015/10/19 16:55:08 Start Go HTTP Server

HTTPレスポンスを確認
$ curl localhost:4000
Hello, World
$ curl localhost:4000/test
TEST

同じサーバをスタートすると。既にポートを使っているのでエラー。
$ go run web.go
2015/10/19 16:55:18 Start Go HTTP Server
2015/10/19 16:55:18 ListenAndServe: listen tcp :4000: bind: address already in use
exit status 1

html/template
GoのHTMLテンプレートにはメジャーどころとして text/templatehtml/template があるらしいです。
ここでは、html/templateパッケージを利用して、テンプレートファイルを読み込んで出力します。こちらの方がセキュアなHTMLを生成するそうです。
テンプレートは、親テンプレートと小テンプレートの2つを用意しました。
現在時刻を表示する単純なウェブページを作ります。
時刻オブジェクトを文字列に変換するときは、fmt.Sprint(hoo)を使いました。
ほかにも、文字・数値変換にはstrconvパッケージも使えます。この場合は、パッケージをインポートして、unixtime := strconv.FormatInt(time.Now().Unix(), 10)というふうに書きます。これはどちらかと言うと、2進数表現にしたい時など、細かい用途に使うのかな?と思いました。
web.go
package main

import (
    "fmt"
    "net/http"
    "log"
    "html/template"
    "time"
)

type TemplateData struct {
    Title string
    Datetime string
    Unixtime string
}

func HelloServer(w http.ResponseWriter, r *http.Request) {
    // 親テンプレートindex.htmlと、小テンプレートbody.htmlを用意
    tmpl := template.Must(template.ParseFiles("views/index.html", "views/body.html"))

    // タイトル
    title := "現在の時刻"

    // bodyに表示するコンテンツ
    // 時刻オブジェクトを文字列に変換
    datetime := fmt.Sprint(time.Now())
    unixtime := fmt.Sprint(time.Now().Unix())

    // テンプレートを実行して出力
    templatedata := TemplateData{title, datetime, unixtime}
    if err := tmpl.ExecuteTemplate(w, "base", templatedata); err != nil {
        fmt.Println(err)
    }
}

func main() {
    http.HandleFunc("/", HelloServer)
    log.Printf("Start Go HTTP Server")
    err := http.ListenAndServe(":4000", nil)
    if err != nil {
       log.Fatal("ListenAndServe: ", err)
    }
}
views/index.html
{{define "base"}}
<html>
    <head>
        <meta charset="utf-8">
        <title>{{.Title}}</title>
    </head>
    <body>
        {{template "body" .}}
    </body>
</html>
{{end}}
views/body.html
{{define "body"}}
    <dev id="content">
        <p>Datetime: {{.Datetime}}</p>
        <p>Unixtime: {{.Unixtime}}</p>
    </dev>
{{end}}
go run
$ go run web.go
2015/10/20 09:57:18 Start Go HTTP Server
$ curl localhost:4000
<html>
    <head>
        <meta charset="utf-8">
        <title>現在の時刻</title>
    </head>
    <body>

    <dev id="content">
        <p>Datetime: 2015-10-20 09:57:19.297331721 &#43;0900 JST</p>
        <p>Unixtime: 1445302639</p>
    </dev>

    </body>
</html>

なにか色々できそうな気がしてきますね。


参考文献)
http - The Go Programming Language
https://golang.org/pkg/net/http/
A Tour of Go Web servers
http://go-tour-jp.appspot.com/#57
Go言語によるwebアプリの作り方
http://www.slideshare.net/yasi_life/goweb-16448500
Codelab: Webアプリケーションを書いてみよう - golang.jp
http://golang.jp/codelab-wiki
テンプレートの処理 | build-web-application-with-golang
https://astaxie.gitbooks.io/build-web-application-with-golang/content/ja/07.4.html
逆引きGolang (日付と時刻)
http://ashitani.jp/golangtips/tips_time.html
Golangでの文字列・数値変換 - 小野マトペの納豆ペペロンチーノ日記
http://matope.hatenablog.com/entry/2014/04/22/101127

Title image from Alyson Hurt
The Go gopher was designed by Renee French. http://reneefrench.blogspot.com/

2015年10月16日金曜日

Go lang "Hello, world" on Mac OS X

This is a sample Go app for "Hello, world".

1) get "Apple OS X" install package from download page.
https://golang.org/dl/

2) Install gox.x.x.darwin-amd64.pkg on your Mac.

3) make code dir and set $GOPATH
$ mkdir ~/gocode
$ echo 'export GOPATH=$HOME/gocode' >> ~/.bashrc
$ source $HOME/.bashrc
4) make "Hello, world."
$ mkdir -p ~/gocode/src/hello
$ cd ~/gocode/src/hello
$ emacs hello.go
package main

import "fmt"

func main() {
 fmt.Printf("Hello, world.\n")
}

$ go install
$ tree ~/gocode
/Users/yako/gocode
├── bin
│   └── hello
└── src
    └── hello
        └── hello.go

3 directories, 2 files

$ $GOPATH/bin/hello
Hello, world.
5) make library.
$ mkdir ~/gocode/src/stringutil
$ cd ~/gocode/src/stringutil
$ emacs reverse.go
// Package stringutil contains utility functions for working with strings.
package stringutil

// Reverse returns its argument string reversed rune-wise left to right.
func Reverse(s string) string {
 r := []rune(s)
 for i, j := 0, len(r)-1; i < len(r)/2; i, j = i+1, j-1 {
  r[i], r[j] = r[j], r[i]
 }
 return string(r)
}

$ cd ~/gocode/src/hello
$ emacs hello.go
package main

import (
       "fmt"

       "stringutil"
)

func main() {
     fmt.Printf(stringutil.Reverse("!oG ,olleH"))
}

$ go install
$ tree ~/gocode
/Users/yako/gocode
├── bin
│   └── hello
├── pkg
│   └── darwin_amd64
│       └── stringutil.a
└── src
    ├── hello
    │   ├── hello.go
    └── stringutil
        └── reverse.go

6 directories, 5 files

$ $GOPATH/bin/hello
Hello, Go!
6) make reverse test.
$ cd ~/gocode/src/stringutil
$ emacs reverse_test.go
package stringutil

import "testing"

func TestReverse(t *testing.T) {
 cases := []struct {
  in, want string
 }{
  {"Hello, world", "dlrow ,olleH"},
  {"Hello, 世界", "界世 ,olleH"},
  {"", ""},
 }
 for _, c := range cases {
  got := Reverse(c.in)
  if got != c.want {
   t.Errorf("Reverse(%q) == %q, want %q", c.in, got, c.want)
  }
 }
}

$ tree ~/gocode
/Users/yako/gocode
├── bin
│   └── hello
├── pkg
│   └── darwin_amd64
│       └── stringutil.a
└── src
    ├── hello
    │   └── hello.go
    └── stringutil
        ├── reverse.go
        └── reverse_test.go

6 directories, 5 files

$ go test stringutil
ok   stringutil 0.006s

See also.

How to Write Go Code
https://golang.org/doc/code.html

Top image from Yuko Honda Today's latte, celebrating Go version 1!
The Go gopher was designed by Renee French. http://reneefrench.blogspot.com/

2015年10月1日木曜日

PHP curl with CURLOPT_CONNECTTIMEOUT_MS

If you want to set timeout of milliseconds with PHP curl_setopt, you have to compile new cURL and re-install PHP.

http://php.net/manual/en/function.curl-setopt.php
CURLOPT_CONNECTTIMEOUT_MS
The number of milliseconds to wait while trying to connect. Use 0 to wait indefinitely. If libcurl is built to use the standard system name resolver, that portion of the connect will still use full-second resolution for timeouts with a minimum timeout allowed of one second.
Added in cURL 7.16.2. Available since PHP 5.2.3.
Environment: CentOS6 with PHP5.6

Install cURL
# cd /var/tmp
# git clone https://github.com/bagder/curl.git
# cd curl
# ./buildconf
# ./configure --enable-threaded-resolver
# make 
# make install
# ./src/curl -V
curl 7.45.0-DEV (x86_64-unknown-linux-gnu) libcurl/7.45.0-DEV OpenSSL/1.0.1e zlib/1.2.3 libidn/1.18
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IDN IPv6 Largefile NTLM NTLM_WB SSL libz UnixSockets
Set ldconfig
# echo '/usr/local/lib' > /etc/ld.so.conf.d/custom-libs.conf
# ldconfig
# ldconfig -p | grep curl
 libcurl.so.4 (libc6,x86-64) => /usr/local/lib/libcurl.so.4
 libcurl.so.4 (libc6,x86-64) => /usr/lib64/libcurl.so.4
 libcurl.so (libc6,x86-64) => /usr/local/lib/libcurl.so
 libcurl.so (libc6,x86-64) => /usr/lib64/libcurl.so
Version up PHP
# yum update --skip-broken --disablerepo=base,updates --enablerepo=remi --enablerepo=remi-php56 php
Re-install php-common
# yum reinstall --skip-broken --disablerepo=base,updates --enablerepo=remi --enablerepo=remi-php56 php-common
# php -i | grep cURL
cURL support => enabled
cURL Information => 7.45.0-DEV
Make Test Script
# vim curl_test.php
<?php

$url = 'http://www.yahoo.co.jp/';
echo $url . "\n";

$option = array(
    CURLOPT_CONNECTTIMEOUT_MS => 500,
    CURLOPT_TIMEOUT           => 500,
    CURLOPT_HEADER            => FALSE,
    CURLOPT_FAILONERROR       => TRUE,
    CURLOPT_RETURNTRANSFER    => TRUE
);

if (!$ch = curl_init()) {
    error_log('[error] curl_init');
    exit;
}

try {
    if (!curl_setopt_array($ch, $option)) {
        throw new Exception('[error] curl_setopt');
    }

    if (!$result = curl_exec($ch)) {
        throw new Exception('[error] curl_exec');
    }

    curl_close($ch);

    echo $result;
} catch (Exception $e) {
    curl_close($ch);
    error_log($e->getMessage());
}
Run Test Script.
# php curl_test.php

Top pic from Francisco Gonzalez Snow Curl


2015年9月28日月曜日

ngxtop for analyze Nginx log

https://github.com/lebinh/ngxtop
ngxtop parses your nginx access log and outputs useful, top-like, metrics of your nginx server. So you can tell what is happening with your server in real-time.
ngxtopは、nginxのログをリアルタイムに解析して、topコマンドのように表示してくれるプログラムです。
オプションをつけたりして工夫すると、Apacheのlogも扱えることができるようです。
sshログインして、どこにアクセスが来ているのかな?というふうに、ぱぱっと見ることができるので、使えるようにしておいて損はしないと思います。

以下、インストールメモです。pipで入るので簡単です。
環境: CentOS 6
# yum install python-setuptools
# easy_install pip
# pip install ngxtop
# ngxtop -h
通常のログフォーマットの他に、以下の様な、LTSVフォーマットのログも解析できます。
ngxtopがnginx.confをパースしてくれるので、とくにオプションの指定は必要ないです。
# cat /etc/nginx/nginx.conf
~~~
    log_format  ltsv  "time:$time_local"
                      "\thost:$remote_addr"
                      "\tforwardedfor:$http_x_forwarded_for"
                      "\treq:$request"
                      "\tstatus:$status"
                      "\tsize:$body_bytes_sent"
                      "\treferer:$http_referer"
                      "\tua:$http_user_agent"
                      "\treqtime:$request_time"
                      "\tapptime:$upstream_response_time"
                      "\tvhost:$host";

    access_log  /var/log/nginx/access.log ltsv;
~~~
ngxtop infoと打つと、どのログを解析するのか見ることができます。
# ngxtop info
nginx configuration file:
  /etc/nginx/nginx.conf
access log file:
  /var/log/nginx/access.log
access log format:
  time:$time_local\thost:$remote_addr\tforwardedfor:$http_x_forwarded_for\treq:$request\tstatus:$status\tsize:$body_bytes_sent\treferer:$http_referer\tua:$http_user_agent\treqtime:$request_time\tapptime:$upstream_response_time\tvhost:$host
available variables:
  body_bytes_sent, host, http_referer, http_user_agent, http_x_forwarded_for, remote_addr, request, request_time, status, time_local, upstream_response_time
違うパスのログファイルを解析するときは、オプションを付ければOKです。
Options:
    -l , --access-log   access log file to parse.

Title pic from Linux Screenshots Log Monitoring Tool - ngxtop