2012年5月9日水曜日

AWS Auto Scaling

※this text from livedoor blog 2011/9/14

AmazonのAuto Scalingを利用してインスタンスの負荷をモニタリング インスタンスを自動で増減させ、ロードバランス配下に設置します ※起動するAMIイメージの準備と、Elastic Load Balancingの設定がされている必要があります 参考文献) http://awsdocs.s3.amazonaws.com/AutoScaling/latest/as-dg.pdf


AutoScaling, CloudWatch APIを設定
※ APIがインストールされていないサーバ向けの記事になります Amazon Linux AMIなどはデフォルトでAPIが使えるので、これを利用する手もあります APIがインストールされているならば、環境変数の設定だけ行なえば良いです


Javaのインストール
# yum install java-1.6.0-openjdk



APIの準備
Auto Scaling Command Line Tool http://aws.amazon.com/developertools/2535
# mkdir /usr/local/ec2-api-tools
# cd /usr/local/ec2-api-tools
# wget http://ec2-downloads.s3.amazonaws.com/AutoScaling-2011-01-01.zip
# unzip AutoScaling-2011-01-01.zip
Amazon CloudWatch Command Line Tool http://aws.amazon.com/developertools/2534
# cd /usr/local/ec2-api-tools
# wget http://ec2-downloads.s3.amazonaws.com/CloudWatch-2010-08-01.zip
# unzip CloudWatch-2010-08-01.zip



API用の証明書ファイルの設定
https://aws-portal.amazon.com/ アカウント ↓ セキュリティ証明書 ↓ 「アクセスキー ID」と「シークレットアクセスキー」を確認
# cd /usr/local/ec2-api-tools/AutoScaling-1.0.39.0
# cp AutoScaling-1.0.39.0/credential-file-path.template ./credential-file
# chmod 600 credential-file
# emacs credential-file
AWSAccessKeyId= # アクセスキー IDを入力
AWSSecretKey= # シークレットアクセスキーを入力



環境変数の設定
証明書をダウンロードし/home/username/aws_config/以下に証明書を設置する
/home/username/aws_config/cert-xxxxx.pem
/home/username/aws_config/pk-xxxxx.pem
# cd
# emacs .bashrc
# Java for AWS
export JAVA_HOME=/usr

# AWS cert
export EC2_PRIVATE_KEY=/home/username/aws_config/pk-xxxxx.pem
export EC2_CERT=/home/username/aws_config/cert-xxxxx.pem

# AWS CREDENTIAL
export AWS_CREDENTIAL_FILE=/usr/local/ec2-api-tools/credential-file

# AWS auto-scaling-api
export AWS_AUTO_SCALING_HOME=/usr/local/ec2-api-tools/AutoScaling-1.0.39.0
export AWS_AUTO_SCALING_URL=https://autoscaling.us-east-1.amazonaws.com
export PATH=$PATH:$AWS_AUTO_SCALING_HOME/bin

# AWS CloudWatch
export AWS_CLOUDWATCH_HOME=/usr/local/ec2-api-tools/CloudWatch-1.0.12.1
export PATH=$PATH:$AWS_CLOUDWATCH_HOME/bin
export EC2_REGION=ap-northeast-1 # Tokyo = ap-northeast-1
# bash



コマンドの一覧を確認
# as-cmd
Command Name                                Description                                            
------------                                -----------                                            
as-create-auto-scaling-group                Create a new auto scaling group                        
as-create-launch-config                     Create a new launch config                             
as-create-or-update-trigger                 Creates a new trigger or updates an existing trigger.  
as-delete-auto-scaling-group                Delete the specified auto scaling group                
as-delete-launch-config                     Delete the specified launch configuration              
as-delete-policy                            Delete the specified policy                            
as-delete-scheduled-action                  Delete the specified scheduled action                  
as-delete-trigger                           Delete a  trigger.                                     
as-describe-adjustment-types                Describes all policy adjustment types.                 
as-describe-auto-scaling-groups             Describes the specified auto scaling group(s)          
as-describe-auto-scaling-instances          Describes the specified auto scaling instance(s)       
as-describe-launch-configs                  Describe the specified launch configurations           
as-describe-metric-collection-types         Describes all metric colle... metric granularity types.
as-describe-policies                        Describes the specified policy/policies                
as-describe-process-types                   Describes all scaling process types.                   
as-describe-scaling-activities              Describe a set of activiti...ties belonging to a group.
as-describe-scheduled-actions               Describes the specified scheduled action(s)            
as-describe-triggers                        Describes a trigger including its internal state.      
as-disable-metrics-collection               Disable collection of AutoScaling group metrics        
as-enable-metrics-collection                Enable collection of AutoScaling group metrics         
as-execute-policy                           Executes the specified policy                          
as-put-scaling-policy                       Creates or updates a scaling policy                    
as-put-scheduled-update-group-action        Creates or updates a scheduled update group action     
as-resume-processes                         Resumes all suspended scal... given auto scaling group.
as-set-desired-capacity                     Set the desired capacity of the auto scaling group     
as-set-instance-health                      Set the health of the instance                         
as-suspend-processes                        Suspends all scaling proce... given auto scaling group.
as-terminate-instance-in-auto-scaling-group Terminate a given instance.                            
as-update-auto-scaling-group                Update specified auto scaling group                    
help                                        
version                                     Prints the version of the CLI tool and the API.        

    For help on a specific command, type ' --help'
# mon-cmd
Command Name                       Description                                            
------------                       -----------                                            
help                               
mon-delete-alarms                  Delete alarms                                          
mon-describe-alarm-history         Describe alarm history                                 
mon-describe-alarms                Describe alarms fully.                                 
mon-describe-alarms-for-metric     Describe all alarms associated with a single metric    
mon-disable-alarm-actions          Disable all actions for a given alarm                  
mon-enable-alarm-actions           Enable all actions for a given alarm                   
mon-get-stats                      Get metric statistics                                  
mon-list-metrics                   List user's metrics                                    
mon-put-metric-alarm               Create a new alarm or update an existing one           
mon-set-alarm-state                Manually set the state of an alarm                     
version                            Prints the version of the CLI tool and the API.        

    For help on a specific command, type ' --help'



Amazon AutoScalingを設定
ここよりAutoScalingの設定をしていきます


Elastic Load Balancerを設定
http://aws.amazon.com/console/ ↓ Amazon Elastic Compute Cloud (EC2) ↓ Region指定 ↓ Load Balancers ↓ Create Load Balancer Availability Zonesの設定をする 作成したLoad Balancerを選択 ↓ Instance ↓ Availability Zones ↓ 全てのAvailability Zonesをチェック ↓ Save


セキュリティグループを設定
http://aws.amazon.com/console/ ↓ Amazon Elastic Compute Cloud (EC2) ↓ Region指定 ↓ Security Groups ↓ Create Security Group 80番など、必要なポートを開けておく


LaunchConfigの設定
起動するインスタンスに関する設定を行なう Key Pair Nameの確認方法 http://aws.amazon.com/console/ ↓ Amazon Elastic Compute Cloud (EC2) ↓ Key Pairs
as-create-launch-config コンフィグ名 --key [Key Pair Name] --image-id AMIのID --group セキュリティグループ --instance-type インスタンスタイプ --region リュージョン
例)
# as-create-launch-config launch-config --key my-key-pair-name --image-id ami-xxxxxxxx --group quick-start-1 --instance-type m1.xx-large --region ap-northeast-1
OK-Created launch config
LaunchConfig確認
# as-describe-launch-configs
LAUNCH-CONFIG  launch-config  ami-xxxxxxxx  m1.xx-large



Auto Scaling Groupの設定
スケーリングするときの基本設定を行なう
# as-create-auto-scaling-group [グループ名] --launch-configuration [LaunchConfigのコンフィグ名] --availability-zones [ゾーン名] --min-size [最小サイズ] --max-size [最大サイズ] --load-balancers [Elastic Load BalancerのLoad Balancer名]
例)
as-create-auto-scaling-group scaling-group --launch-configuration launch-config --availability-zones ap-northeast-1a,ap-northeast-1b --min-size 1 --max-size 5 --load-balancers MyLoadBalancer
OK-Created AutoScalingGroup
成功するとインスタンスが立ち上がるので確認する ※インスタンスが確認できるようになるまで1~2分ほど必要
# as-describe-auto-scaling-groups [グループ名]
例)
# as-describe-auto-scaling-groups scaling-group
AUTO-SCALING-GROUP  scaling-group  launch-config  ap-northeast-1b,ap-northeast-1a  MyLoadBalancer  1  5  1
INSTANCE  i-9497a395  ap-northeast-1b  Pending  Healthy  launch-config
ステータスがPendingからInServiceになったら、サーバにアクセスして起動を確認 ※InServiceになってからも、アクセスできるまでに1~2分ほど待つ必要がある


Auto Scaling Policyの設定
スケーリングするときの動作を設定する インスタンスを増やすときの設定
# as-put-scaling-policy [ポリシー名] --auto-scaling-group [Auto Scaling Groupのグループ名] --adjustment=[変動するインスタンス数の値、1以上の数] --type [適応ポリシー ExactCapacity, ChangeInCapacity, PercentChangeInCapacity] --cooldown [インスタンス増減後、現状維持をする秒数]
例)
# as-put-scaling-policy instance-policy-add --auto-scaling-group scaling-group --adjustment=1 --type ChangeInCapacity --cooldown 60
arn:aws:autoscaling:ap-northeast-1:111111111111:scalingPolicy:aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee:autoScalingGroupName/scaling-group:policyName/instance-policy-add
表示された値をMetric Alarmの設定で使用するのでコピーしておく インスタンスを減らすときの設定
# as-put-scaling-policy [ポリシー名] --auto-scaling-group [Auto Scaling Groupのグループ名] --adjustment=[変動するインスタンス数の値、-1以下の数] --type [適応ポリシー ExactCapacity, ChangeInCapacity, PercentChangeInCapacity] --cooldown [インスタンス増減後、現状維持をする秒数]
例)
# as-put-scaling-policy instance-policy-delete --auto-scaling-group scaling-group --adjustment=-1 --type ChangeInCapacity --cooldown 60
arn:aws:autoscaling:ap-northeast-1:111111111111:scalingPolicy:aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee:autoScalingGroupName/scaling-group:policyName/instance-policy-delete
表示された値をMetric Alarmの設定で使用するのでコピーしておく


Metric Alarmの設定
# mon-put-metric-alarm [アラーム名] \
--comparison-operator [不等号の設定] \
--metric-name [監視項目] \
--namespace "AWS/EC2" \
--period [監視間隔] \
--statistic [測定値の算出方法 SampleCount, Average, Sum, Minimum, Maximum] \
--threshold [監視対象の閾値] \
--evaluation-periods [何回目の閾値オーバーからトリガーが発動するか] \
--dimensions "AutoScalingGroupName=[Auto Scaling Groupのグループ名]" \
--alarm-actions [Auto Scaling Policyの値]
例) インスタンスを増やすときの設定 300秒間の平均CPU使用率が50%より多い場合にインスタンスを増やす
# mon-put-metric-alarm HighCPUAlarm \
--comparison-operator GreaterThanThreshold \
--metric-name CPUUtilization \
--namespace "AWS/EC2" \
--period 300 \
--statistic Average \
--threshold 50 \
--evaluation-periods 1 \
--dimensions "AutoScalingGroupName=scaling-group" \
--alarm-actions arn:aws:autoscaling:ap-northeast-1:111111111111:scalingPolicy:aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee:autoScalingGroupName/scaling-group:policyName/instance-policy-add
インスタンスを減らすときの設定 300秒間の平均CPU使用率が10%より少ない場合にインスタンスを減らす
# mon-put-metric-alarm LowCPUAlarm \
--comparison-operator LessThanThreshold \
--metric-name CPUUtilization \
--namespace "AWS/EC2" \
--period 300 \
--statistic Average \
--threshold 10 \
--evaluation-periods 1 \
--dimensions "AutoScalingGroupName=scaling-group" \
--alarm-actions arn:aws:autoscaling:ap-northeast-1:111111111111:scalingPolicy:aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee:autoScalingGroupName/scaling-group:policyName/instance-policy-delete
Metric AlarmはAWSコンソールからも設定できる http://aws.amazon.com/console/ ↓ Amazon CloudWatch ↓ Region指定 ↓ 該当のアラームを選ぶ ↓ Modify ↓ Define Alarm Thresholdの値を編集 ↓ Continue ↓ Continue ↓ Save Alarm


Auto Scaling Policyの確認
# as-describe-policies
戻り値
SCALING-POLICY  [グループ名]  [ポリシー名]     [adjustmentの値]   [適応ポリシー]  [cooldownの値]  「alarm-actionsの値」
ALARM  [アラーム名] [ポリシー名]
例)
# as-describe-policies
SCALING-POLICY  scaling-group  instance-policy-add     1   ChangeInCapacity  60  arn:aws:autoscaling:ap-northeast-1:111111111111:scalingPolicy:aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee:autoScalingGroupName/scaling-group:policyName/instance-policy-add
ALARM  HighCPUAlarm  instance-policy-add
SCALING-POLICY  scaling-group  instance-policy-delete  -1  ChangeInCapacity  60  arn:aws:autoscaling:ap-northeast-1:111111111111:scalingPolicy:aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee:autoScalingGroupName/scaling-group:policyName/instance-policy-delete
ALARM  LowCPUAlarm  instance-policy-delete



Metric Alarmの確認
# mon-describe-alarms  
戻り値
[アラーム名] [ステータス] [ARM] [namespace] [監視項目] [監視間隔] [監視対象の閾値] [測定値の算出方法] [不等号の設定] [監視対象の閾値]
例)
# mon-describe-alarms  
HighCPUAlarm  OK     arn:aws:autoscalin...instance-policy-add  AWS/EC2  CPUUtilization  300  Average  1  GreaterThanThreshold  50.0
LowCPUAlarm   ALARM  arn:aws:autoscalin...tance-policy-delete  AWS/EC2  CPUUtilization  300  Average  1  LessThanThreshold     10.0



Auto Scalingに関するプロセスを確認
# as-describe-process-types
PROCESS  AZRebalance
PROCESS  AddToLoadBalancer
PROCESS  AlarmNotification
PROCESS  HealthCheck
PROCESS  Launch
PROCESS  ReplaceUnhealthy
PROCESS  ScheduledActions
PROCESS  Terminate



Auto Scalingの通知設定
Amazon Simple Notification Serviceで通知先を設定する http://aws.amazon.com/console/ ↓ Amazon Simple Notification Service (SNS) ↓ Region指定 ↓ Create New Topic ↓ Topic Nameを指定する ↓ 作成したトピックを選択 ↓ Create New Subscription ↓ ProtocolでEmailを選択、Endpointに通知先のemailアドレスを入力 ↓ 10分後くらいに確認メールが届くので、confirm URLを開いて認証する ↓ Publish Topicからテストメールを送る Topic ARNをコピーしておく 通知タイプの確認 起動・起動エラー・削除・削除エラー・テストの5種類
# as-describe-auto-scaling-notification-types --region ap-northeast-1
NOTIFICATION-TYPE  autoscaling:EC2_INSTANCE_LAUNCH
NOTIFICATION-TYPE  autoscaling:EC2_INSTANCE_LAUNCH_ERROR
NOTIFICATION-TYPE  autoscaling:EC2_INSTANCE_TERMINATE
NOTIFICATION-TYPE  autoscaling:EC2_INSTANCE_TERMINATE_ERROR
NOTIFICATION-TYPE  autoscaling:TEST_NOTIFICATION
通知設定
# as-put-notification-configuration [AutoScalingグループ名] --notification-types [通知タイプ] --topic-arn [Topic ARN] --region [リュージョン]
テストメールを送る
# as-put-notification-configuration scaling-group --notification-types autoscaling:TEST_NOTIFICATION --topic-arn arn:aws:sns:ap-northeast-1:111111111111:notification-user --region ap-northeast-1
OK-Put Notification Configuration
それぞれの通知タイプの設定をする
as-put-notification-configuration scaling-group --topic-arn arn:aws:sns:ap-northeast-1:111111111111:notification-user --notification-types autoscaling:EC2_INSTANCE_LAUNCH,autoscaling:EC2_INSTANCE_LAUNCH_ERROR,autoscaling:EC2_INSTANCE_TERMINATE,autoscaling:EC2_INSTANCE_TERMINATE_ERROR
確認
as-describe-notification-configurations [AutoScalingグループ名] -headers
例)
# as-describe-notification-configurations scaling-group -headers

NOTIFICATION-CONFIG  GROUP-NAME       TOPIC-ARN                                          NOTIFICATION-TYPE-NAME
NOTIFICATION-CONFIG  scaling-group    arn:aws:sns:ap-northeast-1:111111111111:notification-user  autoscaling:EC2_INSTANCE_LAUNCH
NOTIFICATION-CONFIG  scaling-group    arn:aws:sns:ap-northeast-1:111111111111:notification-user  autoscaling:EC2_INSTANCE_LAUNCH_ERROR
NOTIFICATION-CONFIG  scaling-group    arn:aws:sns:ap-northeast-1:111111111111:notification-user  autoscaling:EC2_INSTANCE_TERMINATE
NOTIFICATION-CONFIG  scaling-group    arn:aws:sns:ap-northeast-1:111111111111:notification-user  autoscaling:EC2_INSTANCE_TERMINATE_ERROR
削除方法
# as-delete-notification-configuration [グループ名] --topic-arn [arnの値]
例)
# as-delete-notification-configuration scaling-group --topic-arn arn:aws:sns:ap-northeast-1:111111111111:notification-user
    Are you sure you want to delete this notification configuration? [Ny]Y
OK-Deleted Notification Configuration



CloudWatch Alarmsの設定
※Auto Scalingによるインスタンスの増減結果を通知する前に より詳細な情報を通知できるので、合わせて設定しておく コンソールから設定する場合 http://aws.amazon.com/console/ ↓ Amazon CloudWatch ↓ Region指定 ↓ アラーム指定 ↓ Modify ↓ Continue ↓ Define Your Actionsから通知先を設定 ↓ Continue ↓ Save Alarm


負荷テスト
立ち上がっているサーバにログインしてCPU負荷を掛けてテスト ストレステストツールのインストール
# wget http://weather.ou.edu/~apw/projects/stress/stress-1.0.4.tar.gz
# tar zxf stress-1.0.4.tar.gz 
# cd stress-1.0.4
# ./configure
# make
# make install
360秒間CPUに負荷を掛ける
# stress --cpu 8 --timeout 360s
Auto Scalingで立ち上がっているInstanceを確認
# as-describe-auto-scaling-instances --headers



AutoScalingの設定削除
これよりAutoScalingの設定を解除する説明をします


AutoScalingの状態を確認
# as-describe-auto-scaling-groups [グループ名]
例)
# as-describe-auto-scaling-groups scaling-group
AUTO-SCALING-GROUP  scaling-group  launch-config  ap-northeast-1b,ap-northeast-1a  MyLoadBalancer  1  3  1
INSTANCE  i-xxxxxxxx  ap-northeast-1a  InService  Healthy  launch-config



AutoScalingのプロセスを停止
# as-suspend-processes [グループ名]
例)
# as-suspend-processes scaling-group
OK-Processes Suspended



インスタンス増減の値を0に設定
# as-update-auto-scaling-group [グループ名] --min-size 0 --max-size 0 
例)
# as-update-auto-scaling-group scaling-group --min-size 0 --max-size 0 
OK-Updated AutoScalingGroup



起動中のインスタンスをTerminate
# as-terminate-instance-in-auto-scaling-group [インスタンスID] --decrement-desired-capacity
例)
# as-terminate-instance-in-auto-scaling-group i-xxxxxxxx --decrement-desired-capacity
    Are you sure you want to terminate this instance?  [Ny]Y
INSTANCE  edfc6b26-5dde-4dd2-a515-46ffc84f5437  InProgress  "At 2011-09-14T08:18:15Z a user request update of AutoScalingGroup constraints to min: 0, max: 0, desired: 0 changing the desired capacity from 1 to 0.  At 2011-09-14T08:19:26Z instance i-xxxxxxxx was taken out of service in response to a user request, shrinking the capacity from 1 to 0."



AutoScalingの状態を確認
INSTANCEがTerminatingになっているのを確認、完全に削除されるのを待つ
# as-describe-auto-scaling-groups scaling-group
例)
# as-describe-auto-scaling-groups scaling-group
AUTO-SCALING-GROUP  scaling-group  launch-config  ap-northeast-1b,ap-northeast-1a  MyLoadBalancer  0  0  0
INSTANCE  i-xxxxxxxx ap-northeast-1a  Terminating  Healthy  launch-config
SUSPENDED-PROCESS  AZRebalance                        User suspended at 2011-09-14T08:16:37Z  scaling-group
SUSPENDED-PROCESS  HealthCheck                        User suspended at 2011-09-14T08:16:37Z  scaling-group
SUSPENDED-PROCESS  Launch                             User suspended at 2011-09-14T08:16:37Z  scaling-group
SUSPENDED-PROCESS  ScheduledActions                   User suspended at 2011-09-14T08:16:37Z  scaling-group
SUSPENDED-PROCESS  ReplaceUnhealthy                   User suspended at 2011-09-14T08:16:37Z  scaling-group
SUSPENDED-PROCESS  Terminate                          User suspended at 2011-09-14T08:16:37Z  scaling-group
SUSPENDED-PROCESS  AlarmNotification                  User suspended at 2011-09-14T08:16:37Z  scaling-group
SUSPENDED-PROCESS  RemoveFromLoadBalancerLowPriority  User suspended at 2011-09-14T08:16:37Z  scaling-group
SUSPENDED-PROCESS  AddToLoadBalancer                  User suspended at 2011-09-14T08:16:37Z  scaling-group



AutoScalingグループを削除
# as-delete-auto-scaling-group [グループ名]
例)
# as-delete-auto-scaling-group scaling-group
    Are you sure you want to delete this AutoScalingGroup? [Ny]Y
OK-Deleted AutoScalingGroup



LaunchConfigを削除
# as-delete-launch-config [コンフィグ名]
例)
# as-delete-launch-config launch-config
    Are you sure you want to delete this launch configuration? [Ny]Y
OK-Deleted launch configuration



再びAutoScalingの状態を確認
# as-describe-auto-scaling-groups [グループ名]
例)
# as-describe-auto-scaling-groups scaling-group
No AutoScalingGroups found