2015年10月27日火曜日

Go lang hot-deploy with go-server-starter

https://github.com/lestrrat/go-server-starter
The start_server utility is a superdaemon for hot-deploying server programs.
Let's try.
Environment: EC2 AMI ID: Amazon Linux AMI 2015.09 (HVM), SSD Volume Type - ami-9a2fb89a

Go Sample app
Install Go
$ sudo yum -y install golang
$ go version
go version go1.4.2 linux/amd64
$ mkdir ~/gocode
$ echo 'export GOPATH=$HOME/gocode' >> ~/.bashrc
$ source $HOME/.bashrc
Install go-server-starter
# go get github.com/lestrrat/go-server-starter/cmd/start_server
# $GOPATH/bin/start_server --version
0.0.2
# $GOPATH/bin/start_server --help
Make sample web app
# mkdir $GOPATH/src/web/
# cd $GOPATH/src/web/
# emacs sample.go
package main

import (
 "fmt"
 "github.com/lestrrat/go-server-starter/listener"
 "log"
 "net"
 "net/http"
 "os"
)

func hello(w http.ResponseWriter, r *http.Request) {
 fmt.Fprintf(w, "Hello world!")
}

func newHandler() http.Handler {
 mux := http.NewServeMux()
 mux.HandleFunc("/", hello)
 return mux
}

func main() {
 var l net.Listener

 // check environment variable
 if os.Getenv("SERVER_STARTER_PORT") != "" {
  listeners, err := listener.ListenAll()
  if err != nil {
   fmt.Println(err)
   return
  }

  if 0 < len(listeners) {
   l = listeners[0]
  }
 }

 // default port
 if l == nil {
  var err error
  l, err = net.Listen("tcp", fmt.Sprintf(":8080"))

  if err != nil {
   fmt.Println(err)
   return
  }
 }

 log.Printf("Start Go HTTP Server")
 fmt.Println(http.Serve(l, newHandler()))

}

# go install
# $GOPATH/bin/web
2015/10/26 07:36:23 Start Go HTTP Server
# curl localhost:8080
Hello world!

Start app with start_server
# KILL_OLD_DELAY=5 nohup $GOPATH/bin/start_server --port=8000 --pid-file=$GOPATH/start_server.pid -- $GOPATH/bin/web >> $GOPATH/start_server.log &

# cat $GOPATH/start_server.log
starting new worker 26499
2015/10/26 08:36:55 Start Go HTTP Server

$ curl http://127.0.0.1:8000/
Hello world!

Try to hot-deploy.
# cd $GOPATH/src/web/
# emacs sample.go
        fmt.Fprintf(w, "Hello world!")
->
        fmt.Fprintf(w, "Hello world!!!!")
# go install
Graceful restart.
# /bin/cat $GOPATH/start_server.pid | xargs kill -HUP
Check log.
# cat $GOPATH/start_server.log
received HUP (num_old_workers=TODO)
spawning a new worker (num_old_workers=TODO)
starting new worker 2618
2015/10/27 02:43:21 Start Go HTTP Server
new worker is now running, sending TERM to old workers:2576
sleep 5 secs
killing old workers
old worker 2576 died, status:512
Check result.
$ curl http://127.0.0.1:8000/
Hello world!!!!
That's fine.

No error with graceful restart?
Set Nginx for Reverse Proxy
Nginx is necessary for safe connection from the HTTP cliant. Nginx Reverse Proxy makes re-handshake for the backend.
$ sudo yum -y install nginx
$ sudo emacs /etc/nginx/nginx.conf
worker_processes  1;
events {
    worker_connections  50000;
}
http {
    include     mime.types;
    access_log  off;
    upstream app {
        server 127.0.0.1:8000 max_fails=10;
        server localhost:8000 max_fails=10 backup;
    }
    server {
        location / {
            proxy_pass http://app;
            proxy_next_upstream error http_502;
        }
    }
}
$ sudo service nginx start
$ curl http://127.0.0.1/
Hello world!!!!
Run long time ab
$ sudo yum -y install httpd-tools
# ab -c 10 -n 200000 http://127.0.0.1/
Graceful restart
$ /bin/cat $GOPATH/start_server.pid | xargs kill -HUP
$ tail -f  $GOPATH/start_server.log
received HUP (num_old_workers=TODO)
spawning a new worker (num_old_workers=TODO)
starting new worker 2766
2015/10/27 02:56:48 Start Go HTTP Server
new worker is now running, sending TERM to old workers:2759
sleep 5 secs
killing old workers
old worker 2759 died, status:512
Result from ab
Failed requests:        0

Benchmark
start_server support UNIX domain socket. Let's Benchmark Port vs UNIX domain socket.
Instance type: c4.xlarge
Benchmark command.
# ab -c 100 -n 1000000 http://127.0.0.1/
start_server start commands:
Port
# KILL_OLD_DELAY=5 nohup $GOPATH/bin/start_server --port=8000 --pid-file=$GOPATH/start_server.pid -- $GOPATH/bin/web >> $GOPATH/start_server.log &

UNIX domain socket
$ KILL_OLD_DELAY=5 nohup $GOPATH/bin/start_server --path /tmp/app.sock --pid-file=$GOPATH/start_server.pid -- $GOPATH/bin/web >> $GOPATH/start_server.log &
$ chmod 777 /tmp/app.sock
Nginx confs:
Port
    upstream app {
        server 127.0.0.1:8000;
    }

UNIX domain socket
    upstream app {
        server unix:/tmp/app.sock;
    }
Results.
Port
Requests per second:    12544.89 [#/sec] (mean)

UNIX domain socket
Requests per second:    18435.59 [#/sec] (mean)

Tips
How to stop start_server
# /bin/cat $GOPATH/start_server.pid | xargs kill -TERM
# cat $GOPATH/start_server.log
received TERM, sending TERM to all workers:2766
worker 2766 died, status:512
exiting

See also.
Server Starter - a superdaemon to hot-deploy server programs
http://www.slideshare.net/kazuho/server-starter-a-superdaemon-to-hotdeploy-server-programs
Fix panic when SERVER_STARTER_PORT is not defined by handlename · Pull Request #3 · lestrrat/go-server-starter
https://github.com/lestrrat/go-server-starter/pull/3
Docker と SO_REUSEPORT を組み合わせてコンテナのHot Deployにチャレンジ - blog.nomadscafe.jp
http://blog.nomadscafe.jp/2015/01/docker-and-so-reuseport-hot-deploy.html
Server::Starterから学ぶhot deployの仕組み - $shibayu36->blog;
http://blog.shibayu36.org/entry/2012/05/07/201556
Top image from John Lord Starter