vSphere Customization with Cloud-init While Using vRealize Automation 8 or Cloud.

After spending an enormous amount of time, which I think started somewhere in the summer of last year to get vSphere Customization to work with Cloud-init while using vRealize Automation 8 or vRealize Automation Cloud as the automation platform to provision virtual machine deployments and install, configure the applications running on it.

I finally have a workaround that I can say is guaranteed to work every single time, until something better comes along. i.e. Software components like we have it today within the vRA 7.x platform, which is planned to be released for vRA 8.x in Q3-2020 if everything goes as planned.

With some out-of-the-box thinking, I was able to use IP static assignment ( assignment: static ) within the vRA blueprints to leverage the IP Static pool and the network metadata that we define in vRA via Network Profiles for the targeted networks we want to connect to, while using cloud-init with Ubuntu 16.04 and Ubuntu 18.04 for now, but the principle should be the same for other Linux distributions, even though it seems that RHEL is the only OS today that just works provided traditional Guest OS Customization GOSC is set in cloud-init.

Hoping this was worth the time, I am documenting in this blog the step by step instructions on how to prepare your vSphere templates while leveraging cloud-init,  in addition to for your own reference, a list of all the internet available resources that I looked at while doing my research.

I will also have a video added to the blog later that showcases going through the entire template preparation and also demo after that a typical vRA 8 deployment using static IP assignment while leveraging cloud-init to install selected packages per machine component and execute various commands to setup an application.

I still say that this shouldn’t be that hard for our customers to setup and hopefully Software Component like I mentioned would save us all from all this complexity, of-course this is beside the fact that you still can do this via various configuration management tools such as Ansible and puppet which by the way vRealize Automation 8 and cloud integrate with today out-of-the-box.

How it works?

In a high level when the virtual machine first boots up and gets rebooted to be customized due to the dynamic vCenter customization specs that gets created based on the fact we are using the assignment static property ( assignment: static ) within the blueprint code as you see in the screenshot below, I am making sure that during that time, Cloud-init is in a disabled state.

2020-02-15_11-26-33

After the customization reboot the virtual machine once, there is a Cron Job that I created on the template that execute at startup after a 90 sec of sleep which is enough time for the virtual machine to be customized, rebooted and connected to the network without running the Cron Job as of yet. After the initial reboot and pass the 90sec mark now the Cron Job execute a shell script that enables cloud-init and initializes it running all the needed cloud-init modules. ( init, Config and Final)

Note: feel free to increase the 90 sec if you feel you need more time as the virtual machine being customized. 

The End result, the virtual machine is now customized with an updated host-name and an IP from our targeted static IP pool configured for the network its connected to without having to hack the Cloud Config code any further to setup things like the host-name or even configure the network itself, and more importantly without conflicting with cloud-init which what the problem was all alone.

Let’s get started, Eh!

Template Preparation Steps

  • Once the virtual machine is up and running update the list of available packages and install any new available version of these packages that you have to update your template.

sudo apt-get update && sudo apt-get -y upgrade

    • Install Cloud-init for Ubuntu 16.04. Ubuntu 18.04 have cloud-init pre-installed so you can skip this step

sudo apt-get -y install cloud-init

  • Configure OVF as your Datasource, then save and exit

sudo dpkg-reconfigure cloud-init

  • Enable traditional Guest OS Customization GOSC Script by editing /etc/cloud/cloud.cfg file and adding

disable_vmware_customization: true

Note:  Ensure network configuration is not disabled in /etc/cloud/cloud.cfg, by deleting or hashing the following if it exists:

network:
config: disabled

Or that any /etc/cloud/cloud.cfg.d/* configuration files, such as
99-disable-networking-config.cfg with the folloing content “network: {config: disabled}” doesn’t exist.

  • Set Temp not to clear, by editing /usr/lib/tmpfiles.d/tmp.conf  and adding the prefix # to line 11.

#D /tmp 1777 root root –

  • Configure Open-vm-tools to start after dbus.service by editing /lib/systemd/system/open-vmtools.service file and adding the following under the [Unit] section.

After=dbus.service

  • Reduce the raise network interface time to 1 min by editing /etc/systemd/system/network-online.targets.wants/networking.service file and changing: ( This not applicable on Ubuntu 18.04 )

TimeoutStartSec=5min to TimeoutStartSec=1min

  • Disable cloud-init on First Boot and until customization is complete by creating this file /etc/cloud/cloud-init.disabled

sudo touch /etc/cloud/cloud-init.disabled

  • Create a script your_script.sh in a known location that will be called by a Cron Job that will create later to enable and initialize cloud-init after the customization reboot. The script should contain the following commands:
sudo rm -rf /etc/cloud/cloud-init.disabled
sudo cloud-init init
sleep 20
sudo cloud-init modules --mode config
sleep 20
sudo cloud-init modules --mode final
  • Configure the script to be an executable

sudo chmod +x your_script.sh

  • Create a Cron Job that will run after 90 sec of sleep at boot by typing crontab -e and entering the following:

@reboot ( sleep 90 ; sh /Script_path/your_script.sh )

  • Copy the content below for the Template Cleaning script and create your_clean_script.sh. You can replace cloudadmin with your own user that you setup when you installed the Ubuntu OS
#!/bin/bash

# Add usernames to add to /etc/sudoers for passwordless sudo
users=("ubuntu" "cloudadmin")

for user in "${users[@]}"
do
cat /etc/sudoers | grep ^$user
RC=$?
if [ $RC != 0 ]; then
bash -c "echo \"$user ALL=(ALL) NOPASSWD:ALL\" >> /etc/sudoers"
fi
done

#grab Ubuntu Codename
codename="$(lsb_release -c | awk {'print $2}')"


#Stop services for cleanup
service rsyslog stop

#clear audit logs
if [ -f /var/log/audit/audit.log ]; then
cat /dev/null > /var/log/audit/audit.log
fi
if [ -f /var/log/wtmp ]; then
cat /dev/null > /var/log/wtmp
fi
if [ -f /var/log/lastlog ]; then
cat /dev/null > /var/log/lastlog
fi

#cleanup persistent udev rules
if [ -f /etc/udev/rules.d/70-persistent-net.rules ]; then
rm /etc/udev/rules.d/70-persistent-net.rules
fi

#cleanup /tmp directories
rm -rf /tmp/*
rm -rf /var/tmp/*

#cleanup current ssh keys
#rm -f /etc/ssh/ssh_host_*

#cat /dev/null > /etc/hostname

#cleanup apt
apt-get clean

#Clean Machine ID

truncate -s 0 /etc/machine-id
rm /var/lib/dbus/machine-id
ln -s /etc/machine-id /var/lib/dbus/machine-id

#Clean Cloud-init
cloud-init clean --logs --seed

#cleanup shell history
history -w
history -c

  • Configure the Template Cleaning script to be an executable as well

sudo chmod +x your_clean_script.sh

  • Make sure you can switch to user root by editing the fine /etc/ssh/sshd_config and changing PermitRootLogin to yes

PermitRootLogin yes

  • Set a password for root

sudo passwd root

Note: The reason for the above is to be able to execute the clean script with no issues as I personally had issues executing the clean up script with sudo working with the Ubuntu 18.04. you can always revert it back one the cleanup template is executed. 

  • Execute the Template Cleaning Script.

sudo ./Script_path/your_clean_script.sh

  • Shutdown the virtual machine and turn it into a template.

Shutdown -h now

Note : Just be aware that the cron job will also run if you try to update the template for any reason . So make sure if you do pass 90 sec while doing your change is to re-add the /etc/cloud/cloug-init.disabled file and then re-execute the clean up script again before shutting down the template .

if you dont cloud-init will execute on first boot and you will get the vm customization but your cloud config code wont be applied

Click To See It All In Action On my Youtube Channel !

 

Happy Template Building! Please share! 

The End Eh!

Resources:

https://ubuntu.com/engage/cloud-init-whitepaper
https://debconf17.debconf.org/talks/164/
https://cloudinit.readthedocs.io/en/latest/
https://events.linuxfoundation.org/wp-content/uploads/2017/12/Cloud-init-The-cross-cloud-magic-sauce_Smith_moser.pdf
https://www.youtube.com/watch?v=RHVhIWifVqU
https://www.youtube.com/watch?v=y8WA1BUlT-Q
https://linuxtechlab.com/executing-commands-scripts-at-reboot/
https://blogs.vmware.com/management/2019/02/building-a-cas-ready-ubuntu-template-for-vsphere.html
http://kb.vmware.com/s/article/56409
https://kb.vmware.com/s/article/59687
http://kb.vmware.com/s/article/59557
http://kb.vmware.com/s/article/2378666
https://blah.cloud/infrastructure/using-cloud-init-for-vm-templating-on-vsphere/
http://ubuntu.com/blog/cloud-init-v-18-2-cli-subcommands
http://lucd.info/2019/12/06/cloud-init-part-1-the-basics/

 

Blueprinting Cloud Automation Services Cloud-init vRA Blueprints vRealize Automation vSphere Customization