bbpd is a http proxy server for Amazon's DynamoDB service.bbpd uses GoDynamo package for the Amazon DynamoDB database. It made by smugmug.
It supports JSON documents and can post/get data through from HTTP POST method. Oh, that is pretty nice.
So let's try to make DynamoDB HTTP Proxy.
Environment: EC2 AMI ID: Amazon Linux AMI 2015.09 (HVM), SSD Volume Type - ami-9a2fb89a
Install Go
$ sudo yum -y install golang zsh $ go version go version go1.4.2 linux/amd64 $ mkdir ~/gocode $ echo 'export GOPATH=$HOME/gocode' >> ~/.bashrc $ source $HOME/.bashrc
Install bbpd
$ go get github.com/smugmug/bbpd $ sudo cp $GOPATH/bin/bbpd /usr/bin/ $ sudo cp $GOPATH/src/github.com/smugmug/bbpd/bin/bbpd/bbpd_ctl /usr/bin/ $ sudo cp $GOPATH/src/github.com/smugmug/bbpd/bin/bbpd/bbpd_daemon /usr/bin/ $ sudo chmod 755 /usr/bin/bbpd_ctl $ sudo chmod 755 /usr/bin/bbpd_daemonSet config file.
$ cp $GOPATH/src/github.com/smugmug/godynamo/conf_file/test_aws-config.json ~/.aws-config.json
$ emacs ~/.aws-config.json
{
"extends":[],
"services": {
"default_settings":{
"params":{
"access_key_id":"myAccessKey",
"secret_access_key":"mySecret",
"use_sys_log":true
}
},
"dynamo_db": {
"host":"dynamodb.us-east-1.amazonaws.com",
"zone":"us-east-1",
"scheme":"http",
"port":80,
"keepalive":false,
"iam": {
"use_iam":false,
"role_provider":"",
"access_key":"",
"secret_key":"",
"token":"",
"base_dir":"",
"watch":false
}
}
}
}
Start.
$ bbpd_ctl start **** starting bbpd as daemon bbpd - started 2015/10/23 02:22:17 global conf.Vals initialized 2015/10/23 02:22:17 not using iam, assume credentials hardcoded in conf file 2015/10/23 02:22:17 starting bbpd... 2015/10/23 02:22:17 induce panic with ctrl-c (kill -2 2801) or graceful termination with kill -[1,3,15] 2801 2015/10/23 02:22:17 trying to bind to port:12333 2015/10/23 02:22:17 init routing on port 12333 2015/10/23 02:22:17 global conf.Vals initialized 2015/10/23 02:22:17 not using iam, assume credentials hardcoded in conf file 2015/10/23 02:22:17 starting bbpd... 2015/10/23 02:22:17 induce panic with ctrl-c (kill -2 2804) or graceful termination with kill -[1,3,15] 2804 2015/10/23 02:22:17 trying to bind to port:12333 2015/10/23 02:22:17 port 12333 already in use 2015/10/23 02:22:17 trying to bind to port:12334 2015/10/23 02:22:17 init routing on port 12334Check processes.
$ pgrep -l bbpd 24330 bbpd 24333 bbpdGet status.
$ curl -H "X-Bbpd-Indent: true" "http://localhost:12333/Status"
{
"Status": "ready",
"AvailableHandlers": [
"/DescribeTable/",
"/DeleteItem",
"/ListTables",
"/CreateTable",
"/UpdateTable",
"/StatusTable/",
"/PutItem",
"/PutItemJSON",
"/GetItem",
"/GetItemJSON",
"/BatchGetItem",
"/BatchGetItemJSON",
"/BatchWriteItem",
"/BatchWriteItemJSON",
"/UpdateItem",
"/Query",
"/Scan",
"/RawPost/",
"/"
],
"Args": {
"X-Bbpd-Indent": "set '-H \"X-Bbpd-Indent: True\" ' to indent the top-level json",
"X-Bbpd-Verbose": "set '-H \"X-Bbpd-Verbose: True\" ' to get verbose output"
},
"Summary": {
"StartTime": "2015-10-23 08:12:18.875147234 +0000 UTC",
"RunningTime": "11.531750029s",
"LongestResponse": "0.00ms",
"AverageResponse": "0.00ms",
"LastResponse": "no requests made yet",
"ResponseCount": "0"
}
}
Now, bbpd is ready.Run curl
Sample scripts are here. https://github.com/smugmug/bbpd/tree/master/tests
Create table
$ curl -H "X-Amz-Target: DynamoDB_20120810.CreateTable" -X POST -d '
{
"AttributeDefinitions": [
{
"AttributeName": "user_id",
"AttributeType": "S"
}
],
"KeySchema": [
{
"AttributeName": "user_id",
"KeyType": "HASH"
}
],
"ProvisionedThroughput": {
"ReadCapacityUnits": 100,
"WriteCapacityUnits": 100
},
"TableName": "bbpd-test"
}
' http://localhost:12333/
Put
curl -H "X-Amz-Target: DynamoDB_20120810.PutItem" -X POST -d '
{
"Item": {
"num": 1,
"numlist": [
6,
7,
1,
2,
3,
9,
-7234234234.234234
],
"stringlist": [
"pk1_a",
"pk1_b",
"pk1_c"
],
"user_id": "my_user_id1"
},
"TableName": "bbpd-test"
}
' http://localhost:12333/PutItemJSON
Get
$ curl -H "X-Amz-Target: DynamoDB_20120810.GetItem" -X POST -d '
{
"Key": {
"user_id": {
"S": "my_user_id1"
}
},
"TableName": "bbpd-test"
}
' "http://localhost:12333/GetItemJSON"
{"Item":{"num":1,"numlist":[-7.234234234234234e+09,3,9,7,2,1],"stringlist":["pk1_c","pk1_a","pk1_b"],"user_id":"my_user_id1"}}
Looks fine!Check table by SQL using ddbcli
https://github.com/winebarrel/ddbcli
$ gem install ddbcli --no-ri --no-rdoc Fetching: ddbcli-0.5.3.gem (100%)
$ ddbcli -k ACCESS_KEY -s SECRET_KEY -r REGION_OR_ENDPOINT
ap-northeast-1> select all * from bbpd-test;
[
{"num":1,"numlist":[6,3,9,7,2,1,-7234234234.234234],"stringlist":["pk1_a","pk1_b","pk1_c"],"user_id":"my_user_id1"}
]
// 1 row in set (0.02 sec)
That's correct.
Benchmark
DynamoDB Provisioned Read Capacity Units: 10000 Provisioned Write Capacity Units: 10000 EC2 Instance type: c4.large, c4.xlarge, c4.2xlarge
# emacs postdata.file
{
"Key": {
"user_id": {
"S": "my_user_id1"
}
},
"TableName": "bbpd-test"
}
# ab -n 100000 -c 100 -p postdata.file -T "application/json" http://localhost:12333/GetItemJSON c4.large Requests per second: 4754.25 [#/sec] (mean) c4.xlarge Requests per second: 8591.35 [#/sec] (mean) c4.2xlarge Requests per second: 8540.63 [#/sec] (mean) Many RETRY occurs with godynamo authreq.retryReq.
Tips
Set log dir.
$ sudo emacs /usr/bin/bbpd_ctl
LOG=/path/to/log/bbpd.log
start)
echo -n "**** starting bbpd as daemon\n"
$INSTALL_PATH/$DAEMON >> $LOG 2>&1 || true
$INSTALL_PATH/$DAEMON >> $LOG 2>&1 || true
echo "bbpd - started\n"
;;
Amazon DynamoDB logo / Amazon Web Services LLC - http://aws.typepad.com/aws/2011/12/introducing-aws-simple-icons-for-your-architecture-diagrams.html
gopher-side_color.{ai,svg,png} was created by Takuya Ueda (http://u.hinoichi.net). Licensed under the Creative Commons 3.0 Attributions license. - https://github.com/golang-samples/gopher-vector#gopher-side_color
The Go gopher was designed by Renee French. (http://reneefrench.blogspot.com/)
