Installing Chef Server on AWS using Python

This post is about how to install chef server on a AWS EC2 instance using fabric python library.

Create EC2 Instance

In this link you will find all the steps required to create and start an EC2 instance using python

Installing Chef server using Fabric Library

Fabric is a python library and command-line tool that makes easy the use of SSH for application deployment or systems administration tasks.

Fabric installation

    pip install fabric

First Steps with fabric

First at all we must create a file named fabfile.py we will add the code below to this file in order to check the linux box type:

    from fabric.api import run

    def host_type():
        run('uname -s')

We can run fab to see what happens:

    $fab show_host_type  -i ~/.ssh/my_key.pem --host ec2-36-37-35-346.us-west-2.compute.amazonaws.com -u ubuntu
    [ec2-36-37-35-346.us-west-2.compute.amazonaws.com] Executing task 'show_host_type'
    [ec2-36-37-35-346.us-west-2.compute.amazonaws.com] run: uname -s
    [ec2-36-37-35-346.us-west-2.compute.amazonaws.com] out: Linux
    [ec2-36-37-35-346.us-west-2.compute.amazonaws.com] out: 


Done.

Encrypting EC2 Volume

Chef components such as recipes, databags, etc, usually store sensitive data about our infraestructure, for that reason we will encrypt the block storage that we attached on the first step when we created the ec2 instance with python.

In our example we will encrypt /dev/xvdf device using luks library, the first step is setting up the encryption on the device, to do that, we add another task on the fabfile:

    def setup_luks(device):
        cmd  = 'sudo cryptsetup -y --cipher blowfish luksFormat {}'.format(device)
        run(cmd)

fab setup_luks:'/dev/xvdf' -i ~/.ssh/my_key.pem --host ec2-36-37-35-346.us-west-2.compute.amazonaws.com -u ubuntu

[ec2-36-37-35-346.us-west-2.compute.amazonaws.com] Executing task 'setup_luks'
[ec2-36-37-35-346.us-west-2.compute.amazonaws.com] run: sudo cryptsetup -y --cipher blowfish luksFormat /dev/xvdf
[ec2-36-37-35-346.us-west-2.compute.amazonaws.com] out: 
[ec2-36-37-35-346.us-west-2.compute.amazonaws.com] out: WARNING!
[ec2-36-37-35-346.us-west-2.compute.amazonaws.com] out: ========
[ec2-36-37-35-346.us-west-2.compute.amazonaws.com] out: This will overwrite data on /dev/xvdf irrevocably.
[ec2-36-37-35-346.us-west-2.compute.amazonaws.com] out: 
[ec2-36-37-35-346.us-west-2.compute.amazonaws.com] out: Are you sure? (Type uppercase yes): YES
[ec2-36-37-35-346.us-west-2.compute.amazonaws.com] out: Enter passphrase: 
[ec2-36-37-35-346.us-west-2.compute.amazonaws.com] out: Verify passphrase: 
[ec2-36-37-35-346.us-west-2.compute.amazonaws.com] out: 


Done.
Disconnecting from ec2-54-69-12-111.us-west-2.compute.amazonaws.com... done.

The next step is verify the device, in order to do this we will add another task to the fabfile:

    def verify_luks_device(device):
        cmd = 'sudo file -s {0}'.format(device)
        run(cmd)
    fab verify_luks_device:'/dev/xvdf' -i ~/.ssh/my_key.pem --host ec2-36-37-35-346.us-west-2.compute.amazonaws.com -u ubuntu

[ec2-54-69-12-111.us-west-2.compute.amazonaws.com] Executing task 'verify_luks_device'
[ec2-54-69-12-111.us-west-2.compute.amazonaws.com] run: sudo file -s /dev/xvdf
[ec2-54-69-12-111.us-west-2.compute.amazonaws.com] out: /dev/xvdf: LUKS encrypted file, ver 1 [blowfish, cbc-plain, sha1] UUID: e20d71a4-0a4a-43ba-84cc-1ab3470e845f

Now We now that We have a valid luks encryted device, the next is open the device, to do that we add another task to the fabfile:

    def open_luks_device(device, storedev):
        cmd = 'sudo cryptsetup luksOpen {} {}'.format(device, storedev)
        run(cmd)

    fab open_luks_device:'/dev/xvdf,mydevice' -i ~/.ssh/my_key.pem --host ec2-36-37-35-346.us-west-2.compute.amazonaws.com -u ubuntu

[ec2-36-37-35-346.us-west-2.compute.amazonaws.com] Executing task 'open_luks_device'
[ec2-36-37-35-346.us-west-2.compute.amazonaws.com] run: sudo cryptsetup luksOpen /dev/xvdf myvol
[ec2-36-37-35-346.us-west-2.compute.amazonaws.com] out: Enter passphrase for /dev/xvdf: 
[ec2-36-37-35-346.us-west-2.compute.amazonaws.com] out: 

Now we have the device mapped in /dev/mapper/mydevice, the next step is format the device:

    def format_luks_device(storedev):
        run('sudo mkfs.ext4 -m 0 /dev/mapper/{}'.format(storedev))
    fab format_luks_device:'mydevice' -i ~/.ssh/my_key.pem --host ec2-36-37-35-346.us-west-2.compute.amazonaws.com -u ubuntu

[ec2-36-37-35-346.us-west-2.compute.amazonaws.com] Executing task 'format_luks_device'
[ec2-36-37-35-346.us-west-2.compute.amazonaws.com] run: sudo mkfs.ext4 -m 0 /dev/mapper/mydevice
[ec2-36-37-35-346.us-west-2.compute.amazonaws.com] out: mke2fs 1.42.9 (4-Feb-2014)
[ec2-36-37-35-346.us-west-2.compute.amazonaws.com] out: Filesystem label=
[ec2-36-37-35-346.us-west-2.compute.amazonaws.com] out: OS type: Linux
[ec2-36-37-35-346.us-west-2.compute.amazonaws.com] out: Block size=4096 (log=2)
[ec2-36-37-35-346.us-west-2.compute.amazonaws.com] out: Fragment size=4096 (log=2)
[ec2-36-37-35-346.us-west-2.compute.amazonaws.com] out: Stride=0 blocks, Stripe width=0 blocks
[ec2-36-37-35-346.us-west-2.compute.amazonaws.com] out: 655360 inodes, 2620928 blocks
[ec2-36-37-35-346.us-west-2.compute.amazonaws.com] out: 0 blocks (0.00%) reserved for the super user
[ec2-36-37-35-346.us-west-2.compute.amazonaws.com] out: First data block=0
[ec2-36-37-35-346.us-west-2.compute.amazonaws.com] out: Maximum filesystem blocks=2684354560
[ec2-36-37-35-346.us-west-2.compute.amazonaws.com] out: 80 block groups
[ec2-36-37-35-346.us-west-2.compute.amazonaws.com] out: 32768 blocks per group, 32768 fragments per group
[ec2-36-37-35-346.us-west-2.compute.amazonaws.com] out: 8192 inodes per group
[ec2-36-37-35-346.us-west-2.compute.amazonaws.com] out: Superblock backups stored on blocks: 
[ec2-36-37-35-346.us-west-2.compute.amazonaws.com] out:         32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632
[ec2-36-37-35-346.us-west-2.compute.amazonaws.com] out: 
[ec2-36-37-35-346.us-west-2.compute.amazonaws.com] out: Allocating group tables: done                            
[ec2-36-37-35-346.us-west-2.compute.amazonaws.com] out: Writing inode tables: done                            
[ec2-36-37-35-346.us-west-2.compute.amazonaws.com] out: Creating journal (32768 blocks): done
[ec2-36-37-35-346.us-west-2.compute.amazonaws.com] out: Writing superblocks and filesystem accounting information: done 
[ec2-36-37-35-346.us-west-2.compute.amazonaws.com] out: 
[ec2-36-37-35-346.us-west-2.compute.amazonaws.com] out: 

The next step is backup the luks header and store it on our local box, We can use the task below:

    def backup_luks_header_device(device):
        run('sudo cryptsetup luksHeaderBackup --header-backup-file /home/ubuntu/luksbackup {}'.format(device))
        run('sudo chown ubuntu /home/ubuntu/luksbackup')
        get('/home/ubuntu/luksbackup', './')
        run('sudo rm /home/ubuntu/luksbackup')
fab backup_luks_header_device:'/dev/xvdf' -i ~/.ssh/mykey.pem --host ec2-36-37-35-346.us-west-2.compute.amazonaws.com -u ubuntu                                                                                                                                                                         
[ec2-36-37-35-346.us-west-2.compute.amazonaws.com] Executing task 'backup_luks_header_device'                                                                                                   
[ec2-36-37-35-346.us-west-2.compute.amazonaws.com] run: sudo cryptsetup luksHeaderBackup --header-backup-file /home/ubuntu/luksbackup /dev/xvdf                                                 
[ec2-36-37-35-346.us-west-2.compute.amazonaws.com] run: sudo chown ubuntu /home/ubuntu/luksbackup                                                                                               
[ec2-36-37-35-346.us-west-2.compute.amazonaws.com] download: /home/manuel/src/blog/2015/01/chef-aws-python/luksbackup <- /home/ubuntu/luksbackup                                                

[ec2-36-37-35-346.us-west-2.compute.amazonaws.com] run: rm /home/ubuntu/luksbackup

The last step is mount the volume:

    def mount_luks_device(storedev, mount_point, owner):
        run('sudo mkdir -p {}'.format(mount_point))
        run('sudo mount /dev/mapper/{} {}'.format(storedev, mount_point))
        run('sudo chown {} {}'.format(owner, mount_point))
fab mount_luks_device:'mydevice,/secure,ubuntu' -i ~/.ssh/mykey.pem --host ec2-36-37-35-346.us-west-2.compute.amazonaws.com -u ubuntu


[ec2-36-37-35-346.us-west-2.compute.amazonaws.com] Executing task 'mount_luks_device'
[ec2-36-37-35-346.us-west-2.compute.amazonaws.com] run: sudo mkdir -p /secure
[ec2-36-37-35-346.us-west-2.compute.amazonaws.com] run: sudo mount /dev/mapper/mydevice /secure
[ec2-36-37-35-346.us-west-2.compute.amazonaws.com] run: sudo chown ubuntu /secure

If you want to unmount and close the device you must execute the task below:

    def unmount_luks_device(storedev, mount_point):
        run('sudo umount {}'.format(mount_point))
        run('sudo cryptsetup luksClose {}'.format(storedev))
    fab unmount_luks_device:'mydevice,/secure' -i ~/.ssh/mykey.pem --host ec2-36-37-35-346.us-west-2.compute.amazonaws.com -u ubuntu

[ec2-36-37-35-346.us-west-2.compute.amazonaws.com] Executing task 'unmount_luks_device'
[ec2-36-37-35-346.us-west-2.compute.amazonaws.com] run: sudo umount /secure
[ec2-36-37-35-346.us-west-2.compute.amazonaws.com] run: sudo cryptsetup luksClose mydevice

Chef Server Installation

The objective is automatize the steps required to install chef server

    def install_chef_server():
        run('sudo apt-get install ruby')
        run(' wget  https://web-dl.packagecloud.io/chef/stable/packages/ubuntu/trusty/chef-server-core_12.0.1-1_amd64.deb')
        run('sudo dpkg -i /home/ubuntu/chef-server-core_12.0.1-1_amd64.deb')
        run('sudo chef-server-ctl reconfigure')

The fabfile.py can be found at this link

References