>
you're reading...
Eucalyptus, General

Building a Private AWS-compatible Cloud for $400


I wrote a short post earlier in the week titled Eucalyptus Cloud on Ubuntu 13.04 SE. This is the promised follow-up to that post and has since been revised which I will explain below. I often use Ubuntu in development environments due to the community support and large availability of libraries and third-party repositories. Over the years I have utilized Red Hat, Fedora, SUSE, Ubuntu, etc. and for a development environment, it is largely a matter of preference and what you the developer know best – yum versus apt, whatever file structure you are used to, or any other reason you might prefer one over the other.

UPDATE: This post was updated (2013 Aug 03) to include notes regarding a requirement to enable CPU Virtualization in the BIOS, and manual installation of KVM, which was apparently not part of the FastStart installation. Additionally, a sample use case is now provided and a link for “Troubleshooting Eucalyptus” (which I did not write and cannot identify the source but found very useful).

Diagram - Eucalyptus Components

Diagram – Eucalyptus Components

That said there is also a great deal of benefit to ensuring that your development environment matches your production environment as closely as possible. In building a cloud for development use I wanted the OS that I was most comfortable with. After all, the VMs that it will host can leverage whatever OS exactly matches your production environments. Through that process I realized that I was only further complicating my setup and given my time constraints I did not have the time to explore this [Ubuntu 13.04 installation] as much as I would have liked.

Ubuntu Challenges

I opted to use the latest and greatest version of Ubuntu – 13.04 Precise Pangolin. Doing so required that I compile Eucalyptus from source as there were no available repositories that supported Ubuntu. And, unlike the blog I referenced in my initial post, I was using the 64-bit SE version rather than the desktop version. I will not walk through that process as it was tedious and involved a lot of testing, failing, and rebuilding. But, I will provide some comments for anyone else wishing to attempt a similar setup in the future:

1. Eucalyptus Console (or eucalyptus-ui for Ubuntu) must also be compiled from source and is only compatible with Eucalyptus Cloud version 3.2+ due to the authentication method used by the API. The console provides a user interface that has a comfortable web interface for managing your cloud rather than depending only on the command-line API tools. This was important to me for ease of use and knowing that I and others will be making common changes to the environment. For some this is not necessary or perhaps counter intuitive if your goal is to leverage tools such as Chef for building your infrastructure via API calls and testing environment builds prior to migration to Amazon Cloud.

2. Be sure you have a solid understanding of your network and network interfaces – one versus two NIC setups for public and private zones, bridge interfaces, VLANs etc. I understand the networking concepts well but I have only done a little work with bridge interfaces and rarely have had to configure them via command-line myself.

In the end it took me only one day in order to get Eucalyptus running on Ubuntu  (primarily due to the wonderful set of instructions provided by Alin Tomescu and Eucalyptus’ own on-site installation guide). But, I spent an entire additional day attempting to try and get the console to work with Eucalyptus version 3.1, then version 3.2, and I had to revert Ubuntu to 12.04 altogether due to mixing instructions on 13.04 before I realized it. I did finally get the console working only to encounter an issue with Eucalyptus Cluster Controller (CC), which was reporting “NOTREADY” and was unable to find a solution in about four hours. Again, due to my time constraints I cut my losses and changed my approach to the one that finally worked.

Server Challenges

Dell PowerEdge 840 Xeon Dual-Core 3040 1.86GHz 4GB 250GB CD Tower Server w/Video & GbLAN - No Operating System

Dell PowerEdge 840 Xeon Dual-Core 3040 1.86GHz 4GB 250GB CD Tower Server w/Video & GbLAN – No Operating System

As the title of the post indicates my entire solution was dependent upon leveraging four servers that were recently purchased for development use – and they are minimal but at $99 a piece! The four servers were identical refurbished Dell PowerEdge 840 Xeon Dual-Core 3040 1.8GHz 4GB 250GB CD Tower Servers w/Video & GbLAN and no OS. As you can imagine I could not do much with these servers individually due to the CPU limitation but they are dual core and the memory and storage space is enough for a development environment. I felt I could maximize their use by clustering them and running VMs which led to this project. My server challenges? No operating system meant I had to load one, but:

1. I spend yet another day attempting to get PXE boot to work (I had never used this before) and was unsuccessful.

2. I burned a CD (not a DVD) for Ubuntu (13.04 then 12.04) and eventually CentOS 6.

The OS install was not difficult and I only attempted the PXE because at the time I did not have any CDRs laying around at the office. Next, install Eucalyptus:

3. Having never used Eucalyptus much less installed and configured it (it’s a bit complex) I wanted to leverage FastStart. The problem there was that FastStart was only available on DVD and you have to boot from it. These servers do not have a DVD reader, only a basic CD reader.

4. I put the image on a USB thumb drive and burned another CDR for Plop Boot Manager. After ten varying attempts I was unable to get the thumb drive to boot and I spent half of my time trying to learn the menus in Plop because more than 50% of the time they were unreadable characters and I was forced to select the USB option from memory based on location in the menu – each time with a different result but always freezing at some point or returning to the Plop GUI after a quick reboot.

I then came across Silvereye which is FastStart (apparently before it was re-branded) and then the lights came on!

* NOTE: Silvereye is bootable, fits on a CD (versus a DVD), and contains CentOS (no need to install CentOS first or you’ll be replacing it as I did on my first CentOS attempt).

The Solutionwww.centos.org - The Community ENTerprise Operating System

  1. Boot your server into the BIOS and enable CPU Virtualization (usually under “Processor”, ensure your server supports this or you cannot launch instances in your cloud!).

    Screenshot: Enable CPU Virtualization

    Screenshot: Enable CPU Virtualization

  2. Ensure your server is connected to the network and that your network supports DHCP (not required but certainly easier).
  3. Download a Eucalyptus Silverye official ISO. I chose faststart-3.3.0.1.iso (Eucalyptus 3.1, CentOS 6, no [User] Console)
  4. Insert into CD drive and boot.
  5. The CentOS install is straight-forward – follow the prompts and for the most part I simply accepted the defaults. Do however note a few key points: (a) There is one screen with a “Configure Network” button in the lower left corner. You need to select it and explicitly select “Automatically start network” for the interface you choose or you’ll have to execute an “ifup [eth0]” manually after boot before you can access your cloud. (b) Install your nodes first and your front-end last – not required but it will save you time as you cannot finish configuring your front-end until all of your nodes are present. This way you do not have to re-run the configuration or utilize euca_conf in order to register new nodes.
  6. Once the installation completes remove the CD and reboot. It will boot to the console log-in.
  7. Log in using the “root” user and password you entered during the installation.
  8. The first boot will ask you basic questions about your setup in order to configure the node. I selected the defaults for every question (note that I have only one NIC in each server, DHCP, and all on a shared network without a separate VLAN ). This process is the same for both the front-end as well as each node (three in my case).
  9. Install a Hypervisor – in this case, KVM, by executing “yum install kvm” (reference guest OS support here: Guest Support Status – KVM).
  10. Reboot the server.
  11. Once the nodes are configured and running execute “ifconfig” and note the assigned IP addresses for each node – you will need them to configure the front-end.
  12. Follow the same process on the front-end entering each node IP address when prompted. I also opted to go ahead and add a large emi.

At this point you have a running cloud and should able to access the admin interface at https://###.###.###.###:8443/ from any other computer on your network. In my case however I still wanted the user console so I needed to upgrade:

Upgrading Eucalyptus 3.1 to 3.2 and Installing Eucalyptus Console

Eucalyptus Console | Eucalyptus Private AWS-compatible Cloud

Eucalyptus Console | Eucalyptus Private AWS-compatible Cloud

1. Use vim (or whatever command line text editor you prefer) and edit /etc/yum.repos.d/eucalyptus.conf to change “3.1” to “3.2”.

2. Also edit /etc/yum.repos.d/euca2ools.conf and change “2.1.0” to “2.1.3” (per Eucalyptus’ Euca2ools compatability requirements).

3. Upgrade each node with “yum update eucalyptus-nc euca2ools” and accept any additional installations for dependencies as well as any key changes since you updated the yum repositories.

4. On the front-end execute “yum update eucalyptus-cloud eucalyptus-cc euca2ools” and accept the same.

5. To install the console, execute “yum install eucalyptus-console”.

6. Reboot the nodes and verify that the nodes are running again with “service eucalyptus-nc status” after the reboot.

7. Reboot the front-end.

The front-end will take a while to boot back up. You can view the progress in two ways: (1) Press any key when you observe the CentOS loading bar and it will toggle to the text-based detailed view of the boot process. (2) Once OpenSSH is running you can log in as root from another computer and tail /var/log/eucalyptus/cloud-output.log. As part of the reboot Eucalyptus will automatically upgrade your databases and make other changes. It could take 10-15 minutes so be patient and do not interrupt this process as it often appears it is cycling or you are receiving errors. As long as the logs keep outputting data, keep waiting.

When this process completes you will be able to log into the admin console at the URL mentioned previously. The user console will take another 5-10 minutes to start so again, be patient. When it does you can access it at https://###.###.###.###:8888/.

The default user/password are admin/admin and you will be forced to update the password on your first admin log-in. Afterward you can use the same credentials for the user console.

Managing Public IP Addresses

In order to easily access your instances on your local network (via SSH or otherwise) you will need an IP address that is routable by your local network’s DHCP server (in my case a Comcast router/gateway that also acts as a DHCP server).

My router only allows the user to enable/disable the DHCP server and specify a lower as well as an upper range of IP addresses. As such I blocked out a lower range of addresses to allow later assignment of static IP addresses to each node server, and I blocked out an upper range of IP addresses to allow for assignment as public IP addresses to my virtual instances. The allowed range of DHCP address assignments in the gateway is therefore defined as 10.1.10.31 through 10.1.10.149. These are the IP addresses that will be assigned to the standard desktops and other hardware on my local network.

My node controllers have DHCP assigned IP addresses of 10.1.10.28, 10.1.10.29, and 10.1.10.30. These are defined in the /etc/eucalyptus/eucalyptus.conf configuration as follows:

NODES=’10.1.10.28 10.1.10.29 10.1.10.30″
 

Then, I defined the IP address range allowed for public IP address assignment in the same configuration as follows:

VNET_PUBLICIPS=”10.1.10.150-10.1.10.200″
 

Restart the cloud and CC by executing the following commands on the CLC command-line:

service eucalyptus-cloud restart
service eucalyptus-cc restart
 

Once this is complete all new instances you launch will have an IP address within this range and be accessible from your local network. In addition you can allocate “elastic” IP addresses to your virtual instances. Do not forget to open the appropriate ports in your Security Groups!

Converting from DHCP to Static IP Addresses

UPDATE: (2013 Aug 05) Today I reconfigured the CLC and all three nodes in order to use static IP addresses rather than their current DHCP addresses. I already modified the router as explained in the previous section above:

* NOTE: This configuration is specific to my network whereby my CLC and all nodes are operating off of a switch with no VLAN separating it from my local network and I have a gateway/router that assigns DHCP IP addresses to all machines on my network (aside now from the exclusion I set previously). My network is 10.1.10.0 whereas many commercial and home routers default to 192.168.1.0 or otherwise. As such the router is also used as the gateway for DNS resolution. In addition please remember that my CLC, CC, and SC are all hosted on the same server.

Cloud Controller (CLC)

  1. Log into the CLC.
  2. Modify the eth0 interface (assuming that is your default interface) by editing /etc/sysconfig/network-scripts/ifcfg-eth0 as follows:
    1. Change “BOOTPROTO” to “none”
    2. Add “NETWORK=10.1.10.0”
    3. Add “NETMASK=255.255.255.0”
    4. Add “IPADDR=10.1.10.30* Set to the static address accordingly (whatever was previously assigned via DHCP to this CLC)
    5. Add “USERCTL=no”
  3. Then, modify /etc/sysconfig/network to include the gateway by adding “GATEWAY=10.1.10.1”.
  4. Confirm that /etc/resolv.conf contains “nameserver 10.1.10.1” for DNS resolution.
  5. Restart networking by executing “service network restart”.

Node Controller (NC)

* The NC varies slightly as it uses the eth0 network interface via a network bridge, br0.

  1. Log into the NC.
  2. Modify the br0 interface (assuming that is your bridge interface) by editing /etc/sysconfig/network-scripts/ifcfg-br0 as follows:
    1. Change “BOOTPROTO” to “none”
    2. Add “NETWORK=10.1.10.0”
    3. Add “NETMASK=255.255.255.0”
    4. Add “IPADDR=10.1.10.29* Set to the static address accordingly (whatever was previously assigned via DHCP to this NC)
    5. Add “USERCTL=no”
  3. Then, modify /etc/sysconfig/network to include the gateway by adding “GATEWAY=10.1.10.1”.
    Confirm that /etc/resolv.conf contains “nameserver 10.1.10.1” for DNS resolution.
  4. Restart networking by executing “service network restart”.

At this point your cloud is running on static IP addresses without potential to conflict with the DHCP assigned IP addresses since we have blocked out a range of IP addresses in the gateway itself.

Upgrading Eucalyptus 3.2 to 3.3

I also upgraded today from 3.2 to 3.3 (ensure you upgrade through 3.2 first to prevent issues, do not attempt to upgrade directly from 3.1 to 3.3):

1. Use vim (or whatever command line text editor you prefer) and edit /etc/yum.repos.d/eucalyptus.conf to change “3.2” to “3.3”.

2. Also edit /etc/yum.repos.d/euca2ools.conf and change “2.1.3” to “3.0” (per Eucalyptus’ Euca2ools compatability requirements).

3. Upgrade each node with “yum update eucalyptus-nc euca2ools” and accept any additional installations for dependencies as well as any key changes since you updated the yum repositories.

4. On the front-end execute “yum update eucalyptus-cloud eucalyptus-cc euca2ools” and accept the same.

4. Restart the Node Controllers by executing “service eucalyptus-nc” on each node.

7. Restart the CLC, CC, and Console be executing “service eucalyptus-cloud restart; service eucalyptus-cc cleanrestart; service eucalyptus-console restart”.

And, there you have it – upgraded! Eucalyptus 3.3 offers some excellent new features, such as: Elastic Load Balancing, Auto Scaling, CloudWatch Monitoring, Resource Tagging, among others (reference the full list of new features here), as well as some additional features in the User Console. You can also listen to a web broadcast about the new features on Eucalyptus’ website here. There is also a good video about the new console features, which you can watch here:

Follow-up

1. Ideally you will configure a separate VLAN on your network in order to isolate your cloud from the rest of your local area network. I opted instead, for simplicity of installation as well as attempting to mimic most users’ network configurations, to skip this step and isolate them by restricting the DHCP server from serving a range of IP addresses that I assigned to my CLC for use instead.

3. Storage will be minimal with the current configuration – particularly since 250GB is the local storage on each instance and cannot be shared amongst the cloud nodes. I will need to add an NAS or at least an external drive to the front-end where the Storage Cluster (SC) is also installed by default. Eucalyptus can support NAS and SAN storage devices or even directly attached storage (DAS) devices [to the SC directly]. As this would cause a single point of failure, NAS and DAS are not available in HA mode but can be used if you do not have a SAN available at this time.

Now you can log-in and launch your own instance (it will take some time on slow servers like these but rewarding nonetheless). And there you have it – a $400 AWS compatible cloud for development, or maybe just your own R&D. And, hopefully this allows you to configure it in only a couple of hours versus the 3.5 days it took me!

Sample Use Case

Last December (2012) I released a script for an “Automated Attached EBS Volume Backup Solution“, which uses numerous AWS API commands for creating automated snapshots of EBS volumes attached to running EC2 instances. As such it is a perfect use-case for Eucalyptus.

Troubleshooting Eucalyptus

I did not write this but came across it while troubleshooting a few minor issues (I revised the above process since then) but as it was a link to a publicly available Google Docs document and I could not identify the owner, I downloaded a copy as a PDF to make available directly through here and ensure that it does not disappear:

Eucalyptus Troubleshooting Guide

Eucalyptus also provides an excellent and comprehensive library of documentation on their website at:

Eucalyptus 3.1 Documentation

Additional Notes and Lessons Learned

  1. Do not log into the user console with the “eucalyptus” account. It is intended for use in the administrative console only and results on the front-end might not work as expected.
  2. When launching a new instances (if you did not complete the section labeled “Managing Public IP Addresses”), select “Use private IP address only” in the “Advanced Settings” on the last page of the “Launch Instance” process. If you skip this step and you do not have IP addresses allocated for assignment by Eucalyptus then you will receive an error indicating the same.
  3. If an instance is running with a private IP address only, and your cluster/cloud is running behind a switch or router, then you cannot access it directly on your network as it will not know where to route the IP address. Instead you’ll first have to log into a server in the cluster (I have only tested from the CLC and not from any of the NCs).

About christopherjcoleman

Independent IT Consultant. Cloud Expert. United States Navy Veteran. Dedicated. Focused. Driven. I make companies better by developing applications to meet specific business needs on reliable, cost-efficient cloud infrastructure. If the right solution doesn't exist then create it. I have achieved my greatest accomplishments because someone else told me "it's not possible; there is no way to do it" - and now there is.

Discussion

2 thoughts on “Building a Private AWS-compatible Cloud for $400

  1. * NOTE: One day after this post was written, Geeks.com closed down ordering with the following note: “Geeks.com Ordering Is Being Turned Off…”. As such links to the servers I purchased are no longer working but since the basic specifications are noted I am sure you can find something similar on Amazon.com or elsewhere.

    Posted by christopherjcoleman | 2013-08-03, 19:29

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: