Category: Amazon

Setting up EC2 Encrypted File Storage

Setting up EC2 Encrypted File Storage

Step 1: Create an EBS volume

You can do this from the AWS web interface or from the command line, for this example, I’m going to do everything from the command line of the EC2 instance I want to configure.

ec2-create-volume --size 10 -z us-east-1c

Size is in gigabytes, the zone you pick must be the same as the zone your EC2 instance is located in otherwise you won’t be able to attach the volume to your instance if you don’t know what zone you are in use the EC2 metadata query toolYou could also just check it from the AWS web console to find out but that’s cheating and violates my Rule #1 of Cloud Computing: To Know the Cloud is to Know it’s API’s. When you have the time, you should spend some time getting to know the EC2 meta-data API, it’s extremely handy for a lot of things, like getting the run time of your instance for example, but I digress…

Sample output will be something like this (save that vol-XXXXXXXX you will need it later):

VOLUME vol-XXXXXXXX 1 us-east-1c creating 2011-11-26T20:08:15+0000

Step 2: Attach it

Using the vol-XXXXXXXX that you received earlier run the following.

ec2-attach-volume vol-XXXXXXXX -i i-XXXXXXXX -d /dev/sdf

Where i-XXXXXXXX is your instance ID and /dev/sdf is a free device to attach it to. Choose something that is not being used between sdf and sdp, note that the Amazon Linux AMI will likely translate /dev/sdf into /dev/xvdf. If you don’t know your instance ID, use the EC2 metadata tool again or look it up in the AWS web console.

Sample output:

ATTACHMENT vol-XXXXXXXX i-XXXXXXXX /dev/sdf attaching 2011-11-26T20:19:45+0000

 Step 3: Setup the volume for encrypted storage

Now that we have an EBS volume and it’s attached, lets ready it by formatting it for encryption

cryptsetup -y luksFormat /dev/xvdf

Note that I went ahead and changed from /dev/sdf to /dev/xvdf because on the Amazon Linux AMI that’s how your devices will get mapped, just make sure you choose the right device here, once you format your volume, there is no going back. The command will prompt you for confirmation, then a passphrase and then quickly prepare the volume (should take a few seconds).

Sample output:

This will overwrite data on /dev/xvdf irrevocably.

Are you sure? (Type uppercase yes): YES  
Enter LUKS passphrase: ***********  
Verify passphrase: **********

Step 4: Verify the results

You can make sure your new shiny volume is indeed setup for encryption by typing the following:

cryptsetup luksDump /dev/xvdf

Sample output:

LUKS header information for /dev/xvdf

Version:           1  
Cipher name:       aes  
Cipher mode:       cbc-essiv:sha256  
Hash spec:         sha1  
Payload offset:    4096  
MK bits:           256  
MK digest:         xx 22 e1 53 6a 17 hj xx d8 d7 05 55 b7 ee 57 c0  
MK salt:           ec d3 2e 0s f6 e0 05 7e 30 rf xx 76 8d 26 fg 00  
                   c3 kl a0 db xx 68 39 d9 a5 30 31 jk 51 dx 00 c0
MK iterations:     32375  
UUID:              93x0e44x-9x3x-47x5-b6bx-7428x3edcc10

Key Slot 0: ENABLED  
    Iterations:             129600
    Salt:                   xx 30 9d 9b 5e 6b e9 a4 dd g3 fa b6 80 dc 55 ze
                            9c b0 fg x8 11 9c ec 41 94 hf be cj 40 89 k3 fd
    Key material offset:    8
    AF stripes:             4000
Key Slot 1: DISABLED  
Key Slot 2: DISABLED  
Key Slot 3: DISABLED  
Key Slot 4: DISABLED  
Key Slot 5: DISABLED  
Key Slot 6: DISABLED  
Key Slot 7: DISABLED

Step 5: Create a device we can map to

Now that we have an encrypted volume set up, we need to set up a device mapping that we can actually mount, to do so type the following replacing myencfs with something easy for you to remember:

cryptsetup luksOpen /dev/xvdf my_enc_fs

You will be prompted for your password and if everything runs right you will be left with a new device that you can mount or use at /dev/mapper/myencfs (assuming you named your device “myencfs”

Sample output:

Enter passphrase for /dev/xvdf: ***********

Step 6: Format with your filesystem of choice

At this point, the volume is now just like any other newly attached disk volume and needs to be formatted and initialized for use. To format for ext4, a fine choice for a filesystem, type the following

mkfs.ext4 -m 0 /dev/mapper/my_enc_fs

Sample output:

mke2fs 1.41.12 (17-May-2010)  
Filesystem label=  
OS type: Linux  
Block size=4096 (log=2)  
Fragment size=4096 (log=2)  
Stride=0 blocks, Stripe width=0 blocks  
65408 inodes, 261632 blocks  
0 blocks (0.00%) reserved for the super user  
First data block=0  
Maximum filesystem blocks=268435456  
8 block groups  
32768 blocks per group, 32768 fragments per group  
8176 inodes per group  
Superblock backups stored on blocks:  
    32768, 98304, 163840, 229376

Writing inode tables: done  
Creating journal (4096 blocks): done  
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 23 mounts or  
180 days, whichever comes first.  Use tune2fs -c or -i to override.

Step 7: Mount it!

Everything’s been initialized, configured and formatted and your filesystem is now ready for use, this is the moment you have been waiting for, go ahead and create a mount point and mount the volume.

mkdir /encrypted_vol
mount /dev/mapper/my_enc_fs /encrypted_vol

You can now start copying files to /encrypted_vol with the confidence your data is encrypted. In case you are paranoid, you can verify this by running the following

cryptsetup status my_enc_fs

You should see something like the following:

/dev/mapper/my_enc_fs is active and is in use.  
  type:  LUKS1
  cipher:  aes-cbc-essiv:sha256
  keysize: 256 bits
  device:  /dev/sdf
  offset:  4096 sectors
  size:    14752 sectors
  mode:    read/write

Step 8: Disable automount

Your next step should be to disable auto-mounting of dm-crypt/LUKS filesystems to prevent your system from hanging on reboot. To do this you need to add rdNOLUKS to your /etc/grub.conf

Mine looked like this after I finished editing it:

# created by imagebuilder  

title Amazon Linux 2011.09 (  
    root (hd0)
    kernel /boot/vmlinuz- root=LABEL=/ console=hvc0 LANG=en_US.UTF-8 rd_NO_LUKS KEYTABLE=us
    initrd /boot/initramfs-
title Amazon Linux AMI  
    root (hd0)
    kernel /boot/vmlinuz- root=LABEL=/ console=hvc0 rd_NO_LUKS
    initrd /boot/initramfs-

Step 9: You’re done!

That’s it, your system is all set for secure file storage. Just remember to always store your encrypted data in the right place and if you forget, securely delete it after copying into the right place. If you should ever want to unmount and fully detach your encrypted volume, do the following:

umount /encrypted_vol  
cryptsetup luksClose my_enc_fs

Your encrypted volume is now ready to be detached from your instance and re-attached to another one if desired. To re-attach run the following commands (again assuming the volume was mounted to /dev/xvdf, you will be prompted for your passphrase after the first command):

cryptsetup luksOpen /dev/xvdf my_enc_fs
mount /dev/mapper/my_enc_fs /encrypted_vol

You will also need to run these command after you reboot your instance to get access to your data. If you have application data like MySQL DB’s located in your encrypted volume, you should create a script that takes care of mounting your volume and making sure MySQL is running after things are set up which by the way is Rule #17 of the cloud: Automate everything, it’s worth doing once, it’s worth automating.

Credit goes to Erik Peterson who originally wrote this article. You can read his blog here:




Vendor and Cloud lock-in; Good? Bad? Indifferent?

Vendor lock-in, also known as proprietary lock-in or customer lock-in, is when a customer becomes dependent on a vendor for products and services. Thus, the customer is unable to use another vendor without substantial switching costs.

The evolving complexity of data center architectures makes migrating from one product to another difficult and painful regardless of the level of “lock-in.” As with applications, the more integrated an infrastructure solution, architecture and business processes, the less likely it is to be replaced.

The expression “If it isn’t broke, don’t fix it” is commonplace in IT.

I have always touted the anti-vendor lock-in motto. Everything should be Open Source and the End User should have the ability to participate, contribute, consume and modify solutions to fit their specific needs. However is this always the right solution?

Some companies are more limited when it comes to resources. Others are incredibly large and complex making the adoption of Open Source (without support) complicated. Perhaps a customer requires a stable and validated platform to satisfy legal or compliance requirements. If the Vendor they select has a roadmap that matches the companies there might be synergy between the two and thus, Vendor lock-in might be avoided. However, what happens when a Company or Vendor suddenly changes their roadmap?

Most organizations cannot move rapidly between architectures and platform investments (CAPEX) which typically only occur every 3-5 years. If the roadmap deviates there could be problems.

For instance, again let’s assume the customer needs a stable and validated platform to satisfy legal, government or compliance requirements. Would Open Source be a good fit for them or are they better using a Closed Source solution? Do they have the necessary staff to support a truly Open Source solution internally without relying on a Vendor? Would it make sense for them to do this when CAPEX vs OPEXi s compared

The recent trend is for Vendors to develop Open Source solutions; using this as a means to market their Company as “Open” which has become a buzzword. Such terms like Distributed, Cloud, Scale Out, and Pets vs Cattle have also become commonplace in the IT industry.

If a Company or individual makes something Open Source but there is no community adoption or involvement is it really an open Source project? In my opinion, just because I post source code to GitHub doesn’t truthfully translate into a community project. There must be adoption and contribution to add features, fixes, and evolve the solution.

In my experience, the Open Source model works for some and not for others. It all depends on what you are building, who the End User is, regulatory compliance requirements and setting expectations in what you are hoping to achieve. Without setting expectations, milestones and goals it is difficult to guarantee success.

Then comes the other major discussion surrounding Public Cloud and how some also considered it to be the next evolution of Vendor lock-in.

For example, if I deploy my infrastructure in Amazon and then choose to move to Google, Microsoft or Rackspace, is the incompatibility between different Public Clouds then considered lock-in? What about Hybrid Cloud? Where does that fit into this mix?

While there have been some standards put in place such as OVF formats the fact is getting locked into a Public Cloud provider can be just as bad or even worse than being locked into an on-premise architecture or Hybrid Cloud architecture, but it all depends on how the implementation is designed. Moving forward as Public Cloud grows in adoption I think we will see more companies distribute their applications across multiple Public Cloud endpoints and will use common software to manage the various environments. Thus, being a “single pane of glass” view into their infrastructure. Solutions like Cloudforms are trying to help solve these current and frustrating limitations.

Recently, I spoke with someone who mentioned their Company selected OpenStack to prevent Vendor lock-in as it’s truly an Open Source solution. While this is somewhat true, the reality is moving from one OpenStack distribution to another is far from simple. While the API level components and architecture are mostly the same across different distributions the underlying infrastructure can be substantially different. Is that not a type of Vendor lock-in? I believe the term could qualify as “Open Source solution lock-in.”

The next time someone mentions lock-in ask them what they truly mean and what they are honestly afraid of. Is it that they want to participate in the evolution of a solution or product or that they are terrified to admit they have been locked-in to a single Vendor for the foreseeable future?

Is it that they want to participate in the evolution of a solution or product or that they are terrified to admit they have been locked-in to a single Vendor for the foreseeable future?

The future is definitely headed towards Open Source solutions and I think companies such as Red Hat and others will guide the way, providing support and validating these Open Source solutions helping to make them effortless to implement, maintain, and scale.

All one needs to do is look at the largest Software Company in the world, Microsoft, and see how they are aggressively adopting Open Source and Linux. This is a far cry from the Microsoft v1.0 which solely invested in their own Operating System and neglected others such as Linux and Unix.

So, what do you think? Is Vendor lock-in, whether software related, hardware related, Private or Public Cloud, truly a bad thing for companies and End Users or is it a case by case basis?

Cloud Wars – Starring Amazon, Microsoft, Google, Rackspace, Red Hat and OpenStack: The fate of the OS!?

Below is my opinion. Feel free to agree or disagree in the comments or shares but please be respectful to others.

There have been some discussions regarding the Cloud Wars and the current state of the Cloud. One thing I recently participated in was a discussion regarding Microsoft, Red Hat, and Linux distribution adoptions.

Since Microsoft announced the release of their software on Linux platforms, adopted Linux distributions and Linux-based software many people are wondering what this brave new world will look like as we move into the future.

First, we should discuss the elephant in the room. Apple has grown considerably in the desktop market while other companies shares have shrunk. We cannot discount the fact that IOS/OSX are in fact Operating Systems. There are also other desktop/server Operating Systems such as Windows, Chrome, Fedora, CentOS, Ubuntu and other Linux distributions. My apologies for not calling out others as there are far too many to mention. Please feel welcome to mention any overlooked in the comments that you feel I should have included.

The recent partnership between Microsoft and Red Hat has been mutually beneficial and we are seeing more companies that historically ignored Linux now forming alliances with distributions as it has been greatly adopted in the Enterprise. The “battlefield” is now more complex than ever.

Vendors must contend with customers moving to the Public Cloud and adopting “Cloud Centric” application design as they move to a Software as a Service model. In the Cloud, some Operating Systems will erode while others will flourish.

Let’s not forget there is a cost for using specific Operating Systems in the Cloud and other options can be less costly. There are ways to offset this by offering users the ability to bring their own licensing or selecting the de facto Operating System of choice for a Public Cloud. These can be viable options for some and deal breakers for others.

Public Clouds like Azure and Google are still young but they will both mature quickly. Many feel Google may mature faster than others and become a formidable opponent to the current Public Cloud leader which is Amazon.

Some have forgotten that Google was previously in a “Cloud War” of their own when they were competing with Yahoo, Microsoft, Ask, Cuil, Snap, Live Search, Excite and many others. The most recent statistics show Google holding at 67.7% of the search market, which is a considerable lead over everyone else. Google after all was born in the Cloud, lives in the Cloud and understands it better than anyone else. Many things they touch turn to gold, like Chrome, Gmail, Android and other web based applications.

In the Private Cloud, Microsoft, VMware, Red Hat, Canonical, and Oracle are in contention with one another. Some are forming strategic alliances and partnerships for the greater good and pushing the evolution of software. Others are ignoring evolution and preferring to move forward, business as usual.

When market shares erode companies sometimes rush and make poor miscalculated decisions. One only needs to look at the fate of Blackberry to see how a company can fall rapidly from the top to the bottom of the market. Last I checked, Blackberry didn’t even own 1% of the market in the Mobile arena.

As we move into the future of Cloud, whether that is Public or Private, we will see more strategic partnerships and barriers collapse. With so many emerging technologies on the horizon and the Operating Systems becoming more of a platform for containerized applications it’s also becoming less relevant than previous.

I have heard individuals predict in the future we will write code directly to the Cloud and I agree that this will eventually happen. Everything will be ambiguous to the developer or programmer and there will be “one ring to rule them all” but the question to be answered is what ring will that be and who will be wearing it?

It’s doubtful we will ever only have a single Cloud, platform or programming language but I think we will see the birth of code and platform translators. I look at computers, technology and programming the same as spoken language. Some people learn a native language only, others learn a native tongue and branch out to other languages and may even one day become a philologist for example.

I am anxious to see how things evolve and am looking forward to seeing the development of the Cloud and internet applications. I hope I am able to witness things such as self-drivings cars, self-piloting airplanes, and real-time data analysis.

Perhaps instead of Cloud we should use the term Galaxy.