ec2 facts¶

Dynamic inventory provides metadata from ec2 module and hostvars contains additional information like:

{u’ansible_all_ipv4_addresses’: [u‘172.30.3.139’],

u’ansible_all_ipv6_addresses’: [u’fe80::4b9:ffff:fe81:65f’], u’ansible_architecture’: u’x86_64’, u’ansible_bios_date’: u‘05/12/2016’, u’ansible_bios_version’: u‘4.2.amazon’, ‘ansible_check_mode’: False, u’ansible_cmdline’: {u’BOOT_IMAGE’: u’/boot/vmlinuz-3.13.0-91-generic’,

u’ansible_date_time’: {u’date’: u‘2016-09-16’,: u’day’: u‘16’, u’epoch’: u‘1473991427’, u’hour’: u‘02’, u’iso8601’: u‘2016-09-16T02:03:47Z’, u’iso8601_basic’: u‘20160916T020347397930’, u’iso8601_basic_short’: u‘20160916T020347’, u’iso8601_micro’: u‘2016-09-16T02:03:47.398008Z’, u’minute’: u‘03’, u’month’: u‘09’, u’second’: u‘47’, u’time’: u‘02:03:47’, u’tz’: u’UTC’, u’tz_offset’: u’+0000’, u’weekday’: u’Friday’, u’weekday_number’: u‘5’, u’weeknumber’: u‘37’, u’year’: u‘2016’},
u’ansible_default_ipv4’: {u’address’: u‘172.30.3.139’,: u’alias’: u’eth0’, u’broadcast’: u‘172.30.3.255’, u’gateway’: u‘172.30.3.1’, u’interface’: u’eth0’, u’macaddress’: u‘06:b9:ff:81:06:5f’, u’mtu’: 9001, u’netmask’: u‘255.255.255.0’, u’network’: u‘172.30.3.0’, u’type’: u’ether’},

u’ansible_default_ipv6’: {}, u’ansible_devices’: {u’xvda’: {u’holders’: [],

u’ansible_distribution’: u’Ubuntu’, u’ansible_distribution_major_version’: u‘14’, u’ansible_distribution_release’: u’trusty’, u’ansible_distribution_version’: u‘14.04’, u’ansible_dns’: {u’nameservers’: [u‘172.30.0.2’],

u’ansible_domain’: u’‘, u’ansible_env’: {u’HOME’: u’/home/ubuntu’,

u’ansible_eth0’: {u’active’: True,

u’device’: u’eth0’, u’ipv4’: {u’address’: u‘172.30.3.139’,

u’ipv6’: [{u’address’: u’fe80::4b9:ffff:fe81:65f’,: u’prefix’: u‘64’, u’scope’: u’link’}],

u’macaddress’: u‘06:b9:ff:81:06:5f’, u’mtu’: 9001, u’pciid’: u’vif-0’, u’promisc’: False, u’type’: u’ether’},

u’ansible_fips’: False, u’ansible_form_factor’: u’Other’, u’ansible_fqdn’: u’ip-172-30-3-139’, u’ansible_gather_subset’: [u’hardware’, u’network’, u’virtual’], u’ansible_hostname’: u’ip-172-30-3-139’, u’ansible_interfaces’: [u’lo’, u’eth0’], u’ansible_kernel’: u‘3.13.0-91-generic’, u’ansible_lo’: {u’active’: True,

u’ansible_lsb’: {u’codename’: u’trusty’,: u’description’: u’Ubuntu 14.04.4 LTS’, u’id’: u’Ubuntu’, u’major_release’: u‘14’, u’release’: u‘14.04’},

u’ansible_machine’: u’x86_64’, u’ansible_machine_id’: u’dd1bac9cb23792a632bd6ee157719517’, u’ansible_memfree_mb’: 715, u’ansible_memory_mb’: {u’nocache’: {u’free’: 932, u’used’: 60},

u’ansible_memtotal_mb’: 992, u’ansible_mounts’: [{u’device’: u’/dev/xvda1’,

u’ansible_nodename’: u’ip-172-30-3-139’, u’ansible_os_family’: u’Debian’, u’ansible_pkg_mgr’: u’apt’, u’ansible_processor’: [u’GenuineIntel’,

u’ansible_processor_cores’: 1, u’ansible_processor_count’: 1, u’ansible_processor_threads_per_core’: 1, u’ansible_processor_vcpus’: 1, u’ansible_product_name’: u’HVM domU’, u’ansible_product_serial’: u’NA’, u’ansible_product_uuid’: u’NA’, u’ansible_product_version’: u‘4.2.amazon’, u’ansible_python’: {u’executable’: u’/usr/bin/python’,

u’ansible_python_version’: u‘2.7.6’, u’ansible_selinux’: False, u’ansible_service_mgr’: u’upstart’, u’ansible_ssh_host’: u‘52.23.213.103’, u’ansible_ssh_host_key_dsa_public’: u’AAAAB3NzaC1kc3MAAACBAKosQQsY22MWjQv/bF7V6kewbM5Z7NIPjYgNL/j8lbUrmZv/kZSAArfs/ByKjC2gesSZUi3hZ1Lo1LmFZjcu00IUKosKnPhHF6hNoyApxHNL1FovWULIWAyAbxpsRNkBNJYQ9Ox0Sklgf1PC94GJwL4SD3lSm180YsaEIw+AC5kJAAAAFQCdQN2qo70bFXxFjSUa0mZ8ORgRSwAAAIBR+lfWXZhjvi5RCccs4xwfrsTq+ujsVsb3qRP3vE+Caxyd8HrdeTDV9jRqJEvsSmKU57L58NBiteYFHfdA0D+HOQm56ZvDEhz/FVRR4ei2UZC04TEsZ4aXfvGzt6OnVRcguiozjeWnglWhikvcdrQHxszRziiaDdubnomlfn0xMQAAAIEAgKs1pwbh9S/dE5nbSqwlkTiOR6jkcPhP8XGSvjcjXqbm+ymnjJ9W9hM7DiaFm83YjWf+3+LOLMtULf7w91xOC0HLRD5M2WWQ/RxATcLN37jhbAs3XkYqZNCtVWOGm6m/wbOiWt3R3SYS5h6Bp4oqHyTYifQsfY+KVSyb29RjZzc=’, u’ansible_ssh_host_key_ecdsa_public’: u’AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCjuapwx2AgG834ti6pGu7gbF2YJmoQ1r6wGVNQrnVHs7rZ5S+hhTSjRmZQ7d5zYgbqFX/f9WGAQgAn89dWyHPE=’, u’ansible_ssh_host_key_ed25519_public’: u’AAAAC3NzaC1lZDI1NTE5AAAAIEOzako99Tn5mO/vjk+ctHc9HKmtdsdJxTqqNIoo0RI3’, u’ansible_ssh_host_key_rsa_public’: u’AAAAB3NzaC1yc2EAAAADAQABAAABAQC7elCMaVXdqIuBrO+4Fx5oPggQdI7JRKv176NsoRtwsYPtxp07t4mllkH6ATsQgkj9wDHOxMk73zyEFaMZskCVNwvN1bwGIyZX/i8JXJtzAdvZZNn1cH1zCi1WD3bRx3dp+p8JU+OmVBhaElPYqm3W+BDWGC7b4YqHbqTHDNTUswgvvLuXr0Qw7fdWMNxb8Va7c3GOrMEkMWnHpMpVCu6Y8zIHdcmJWSMeBXOUKPkXXsaI6jAX8koi8pAFeX95rHbwhP2uo76ueJqWJhDPLm6FdXW+hsPIKDn4eFIhVrHNROml4Z3pAR6FmfidJEi5dvXmefLC6IQH9eDgf7Dn5fS9’, u’ansible_swapfree_mb’: 0, u’ansible_swaptotal_mb’: 0, u’ansible_system’: u’Linux’, u’ansible_system_capabilities’: [u’‘], u’ansible_system_capabilities_enforced’: u’True’, u’ansible_system_vendor’: u’Xen’, u’ansible_uptime_seconds’: 21477, u’ansible_user_dir’: u’/home/ubuntu’, u’ansible_user_gecos’: u’Ubuntu’, u’ansible_user_gid’: 1000, u’ansible_user_id’: u’ubuntu’, u’ansible_user_shell’: u’/bin/bash’, u’ansible_user_uid’: 1000, u’ansible_userspace_architecture’: u’x86_64’, u’ansible_userspace_bits’: u‘64’, ‘ansible_version’: {‘full’: ‘2.1.1.0’,

u’ansible_virtualization_role’: u’guest’, u’ansible_virtualization_type’: u’xen’, u’ec2__in_monitoring_element’: False, u’ec2_ami_launch_index’: u‘0’, u’ec2_architecture’: u’x86_64’, u’ec2_client_token’: u’‘, u’ec2_dns_name’: u’‘, u’ec2_ebs_optimized’: False, u’ec2_eventsSet’: u’‘, u’ec2_group_name’: u’‘, u’ec2_hypervisor’: u’xen’, u’ec2_id’: u’i-435014bb’, u’ec2_image_id’: u’ami-2d39803a’, u’ec2_instance_profile’: u’‘, u’ec2_instance_type’: u’t2.micro’, u’ec2_ip_address’: u‘52.23.213.103’, u’ec2_item’: u’‘, u’ec2_kernel’: u’‘, u’ec2_key_name’: u’albert’, u’ec2_launch_time’: u‘2016-09-15T20:05:34.000Z’, u’ec2_monitored’: False, u’ec2_monitoring’: u’‘, u’ec2_monitoring_state’: u’disabled’, u’ec2_persistent’: False, u’ec2_placement’: u’us-east-1a’, u’ec2_platform’: u’‘, u’ec2_previous_state’: u’‘, u’ec2_previous_state_code’: 0, u’ec2_private_dns_name’: u’ip-172-30-3-139.ec2.internal’, u’ec2_private_ip_address’: u‘172.30.3.139’, u’ec2_public_dns_name’: u’‘, u’ec2_ramdisk’: u’‘, u’ec2_reason’: u’‘, u’ec2_region’: u’us-east-1’, u’ec2_requester_id’: u’‘, u’ec2_root_device_name’: u’/dev/sda1’, u’ec2_root_device_type’: u’ebs’, u’ec2_security_group_ids’: u’sg-69ebc60c,sg-115b376b’, u’ec2_security_group_names’: u’default,abds_secgroup’, u’ec2_sourceDestCheck’: u’true’, u’ec2_spot_instance_request_id’: u’‘, u’ec2_state’: u’running’, u’ec2_state_code’: 16, u’ec2_state_reason’: u’‘, u’ec2_subnet_id’: u’subnet-719a774d’, u’ec2_tag_datanodes’: u’True’, u’ec2_tag_frontendnodes’: u’True’, u’ec2_tag_hadoopnodes’: u’True’, u’ec2_tag_historyservernodes’: u’True’, u’ec2_tag_journalnodes’: u’True’, u’ec2_tag_namenodes’: u’True’, u’ec2_tag_resourcemanagernodes’: u’True’, u’ec2_tag_zookeepernodes’: u’True’, u’ec2_virtualization_type’: u’hvm’, u’ec2_vpc_id’: u’vpc-e6c17c83’, ‘group_names’: [u’ami_2d39803a’,

‘groups’: {‘all’: [u‘52.23.213.103’, u‘54.196.41.145’, u‘54.209.137.235’],

u’ami_2d39803a’: [u‘52.23.213.103’,: u‘54.196.41.145’, u‘54.209.137.235’],

u’ec2’: [u‘52.23.213.103’, u‘54.196.41.145’, u‘54.209.137.235’], u’i-435014bb’: [u‘52.23.213.103’], u’i-e250141a’: [u‘54.196.41.145’], u’i-e350141b’: [u‘54.209.137.235’], u’key_albert’: [u‘52.23.213.103’,

u’security_group_abds_secgroup’: [u‘52.23.213.103’,: u‘54.196.41.145’, u‘54.209.137.235’],
u’security_group_default’: [u‘52.23.213.103’,: u‘54.196.41.145’, u‘54.209.137.235’],
u’tag_datanodes_True’: [u‘52.23.213.103’,: u‘54.196.41.145’, u‘54.209.137.235’],

u’tag_frontendnodes_False’: [u‘54.196.41.145’, u‘54.209.137.235’], u’tag_frontendnodes_True’: [u‘52.23.213.103’], u’tag_hadoopnodes_True’: [u‘52.23.213.103’,

u’tag_historyservernodes_False’: [u‘54.196.41.145’,: u‘54.209.137.235’],

u’tag_historyservernodes_True’: [u‘52.23.213.103’], u’tag_journalnodes_True’: [u‘52.23.213.103’,

u’tag_namenodes_True’: [u‘52.23.213.103’,: u‘54.196.41.145’, u‘54.209.137.235’],
u’tag_resourcemanagernodes_True’: [u‘52.23.213.103’,: u‘54.196.41.145’, u‘54.209.137.235’],
u’tag_zookeepernodes_True’: [u‘52.23.213.103’,: u‘54.196.41.145’, u‘54.209.137.235’],
u’type_t2_micro’: [u‘52.23.213.103’,: u‘54.196.41.145’, u‘54.209.137.235’],

‘ungrouped’: [‘localhost’], u’us-east-1’: [u‘52.23.213.103’,

u’us-east-1a’: [u‘52.23.213.103’,: u‘54.196.41.145’, u‘54.209.137.235’],
u’vpc_id_vpc_e6c17c83’: [u‘52.23.213.103’,: u‘54.196.41.145’, u‘54.209.137.235’]},

‘inventory_dir’: ‘/home/albert/git/aws-cloudformation-by-ansible’, ‘inventory_file’: ‘ec2.py’, ‘inventory_hostname’: u‘52.23.213.103’, ‘inventory_hostname_short’: u‘52’, u’module_setup’: True, ‘omit’: ‘__omit_place_holder__943b1cb4d91b305bcddb59adcbdfd6146fda5e7e’, ‘playbook_dir’: u’/home/albert/git/aws-cloudformation-by-ansible’}

Documentation about ec2 facts: http://docs.ansible.com/ansible/ec2_facts_module.html

In Practice¶

boot.yml launches ec2 instances and ec2.py --list returns running ec2 instances with public ip addresses. When the actual playbook site.yml runs for Big Data Stacks, inventory groups and IPs are required to connect.

Traditional Inventory file¶

inventory.txt contains, for example:

[namenodes]
mycluster0
mycluster1

[resourcemanagernodes]
mycluster0
mycluster1

[datanodes]
mycluster0
mycluster1
mycluster2

[zookeepernodes]
mycluster0
mycluster1
mycluster2

[hadoopnodes]
mycluster0
mycluster1
mycluster2

[historyservernodes]
mycluster2

[journalnodes]
mycluster2
mycluster1
mycluster0

[frontendnodes]
mycluster2

and the following directories for each hostname, e.g. mycluster[0-2] look like:

$ cat host_vars/mycluster0
ansible_ssh_host: 192.168.1.100
zookeeper_id: 0

Dynamic Inventory (ec2)¶

Let’s assume the official ec2.py is used for dynamic inventory for Amazon EC2. (It’s from https://raw.github.com/ansible/ansible/devel/contrib/inventory/ec2.py)

It returns JSON based on running instances on Amazon, for example, if no vm is running:

$ ./ec2.py --list
{
  "_meta": {
    "hostvars": {}
   }
}

Note

chmod a+x ec2.py for executable

If there is running VMs, the return JSON data looks like:

{

“_meta”: {

“hostvars”: {

“54.85.130.171”: {: “ansible_ssh_host”: “54.85.130.171”, “ec2__in_monitoring_element”: false, “ec2_ami_launch_index”: “0”, “ec2_architecture”: “x86_64”, “ec2_client_token”: “”, “ec2_dns_name”: “”, “ec2_ebs_optimized”: false, “ec2_eventsSet”: “”, “ec2_group_name”: “”, “ec2_hypervisor”: “xen”, “ec2_id”: “i-1a2c68e2”, “ec2_image_id”: “ami-2d39803a”, “ec2_instance_profile”: “”, “ec2_instance_type”: “t2.micro”, “ec2_ip_address”: “54.85.130.171”, “ec2_item”: “”, “ec2_kernel”: “”, “ec2_key_name”: “albert”, “ec2_launch_time”: “2016-09-15T17:18:36.000Z”, “ec2_monitored”: false, “ec2_monitoring”: “”, “ec2_monitoring_state”: “disabled”, “ec2_persistent”: false, “ec2_placement”: “us-east-1a”, “ec2_platform”: “”, “ec2_previous_state”: “”, “ec2_previous_state_code”: 0, “ec2_private_dns_name”: “ip-172-30-3-104.ec2.internal”, “ec2_private_ip_address”: “172.30.3.104”, “ec2_public_dns_name”: “”, “ec2_ramdisk”: “”, “ec2_reason”: “”, “ec2_region”: “us-east-1”, “ec2_requester_id”: “”, “ec2_root_device_name”: “/dev/sda1”, “ec2_root_device_type”: “ebs”, “ec2_security_group_ids”: “sg-69ebc60c,sg-115b376b”, “ec2_security_group_names”: “default,abds_secgroup”, “ec2_sourceDestCheck”: “true”, “ec2_spot_instance_request_id”: “”, “ec2_state”: “running”, “ec2_state_code”: 16, “ec2_state_reason”: “”, “ec2_subnet_id”: “subnet-719a774d”, “ec2_tag_datanodes”: “True”, “ec2_tag_frontendnodes”: “False”, “ec2_tag_hadoopnodes”: “True”, “ec2_tag_historyservernodes”: “False”, “ec2_tag_journalnodes”: “True”, “ec2_tag_namenodes”: “True”, “ec2_tag_resourcemanagernodes”: “True”, “ec2_tag_zookeepernodes”: “True”, “ec2_virtualization_type”: “hvm”, “ec2_vpc_id”: “vpc-e6c17c83”

}

}, “ami_2d39803a”: [

], “ec2”: [

], “i-1a2c68e2”: [

], “key_albert”: [

], “security_group_abds_secgroup”: [

], “security_group_default”: [

], “tag_datanodes_True”: [

], “tag_frontendnodes_False”: [

], “tag_hadoopnodes_True”: [

], “tag_historyservernodes_False”: [

], “tag_journalnodes_True”: [

], “tag_namenodes_True”: [

], “tag_resourcemanagernodes_True”: [

], “tag_zookeepernodes_True”: [

], “type_t2_micro”: [

], “us-east-1”: [

], “us-east-1a”: [

], “vpc_id_vpc_e6c17c83”: [

]

}

EC2 Tags for Inventory Groups¶

If you noticed that there are tags for host groups like tag_hadoopnodes_True, ec2 instances have key:value tags and these are used for identifying inventory groups.

Ansible ec2 module provides instance_tags option to define them.

Example:

ec2:
  ...
  instance_tags:
     namenodes: True
     ...
     frontendnodes: False

boot.yml defines tags information when it starts EC2 instances.

add_host¶

This task converts tag_*_True list to Inventory groups therefore existing roles run without changes between dynamic and static file inventories.

convert.yml contains add_host tasks to convert the inventory groups.

For example, to convert hadoopnodes:

- name: convert EC2 tags_*_True inventory groups
  hosts: "tag_hadoopnodes_True"
  tasks:

    - name: Add new instance to host group (hadoopnodes)
      add_host: hostname="{{ hostvars[item]['ec2_ip_address'] }}" groupname=hadoopnodes
      with_items: "{{ groups['tag_hadoopnodes_True'] }}"

The hosts in the tag_hadoopnodes_True list run the add_host task to add its ec2_ip_address (in this case, AWS VPC, virtual private cloud, associates with a floating ip address and it is visible via ec2_ip_address) to the new group name hadoopnodes which the following ansible roles for BigData Stack requires.

site.yml with Include¶

Three Ansible playbooks are provided to deploy virtual clusters on Amazon EC2.

boot.yml: to start EC2 instances
convert.yml: to provide inventory groups from dynamic inventory of ec2.
example-project-nist-fingerprint-matching.yml: to run SDS playbooks