After spending an enormous amount of time, which I think started somewhere in the summer of last year to get vSphere Customization to work with Cloud-init while using vRealize Automation 8 or vRealize Automation Cloud as the automation platform to provision virtual machine deployments and install, configure the applications running on it.

I finally have a workaround that I can say is guaranteed to work every single time, until something better comes along that would help with the vSphere customization and cloud-init conflict during startup.

With some out-of-the-box thinking, I was able to use IP static assignment ( assignment: static ) within the vRA blueprints to leverage the IP Static pool and the network metadata that we define in vRA via Network Profiles for the targeted networks we want to connect to, while using cloud-init with Ubuntu 16.04, Ubuntu 18.04 and Ubuntu 20.04 for now, but the principle should be the same for other Linux distributions, even though it seems that RHEL is the only OS today that just works provided traditional Guest OS Customization GOSC is being set in cloud-init.

Update ( 26/04/2022)  If you trying to use cloud-init with Ubuntu 20 .. Please be aware of this KB as without its resolution, cloud-init will not be able to use the OVF as a datasource therefore userdata will not be passed to the VM when using Cloud-Config in vRealize Automation VMware Cloud Template.

Note: The will also work if you were to use DHCP IP Assignment.

Hoping this was worth the time, I am documenting in this blog the step by step instructions on how to prepare your vSphere templates while leveraging cloud-init,  in addition to for your own reference, a list of all the internet available resources that I looked at while doing my research.

I will also have a video added to the blog later that showcases going through the entire template preparation and also demo after that a typical vRA 8 deployment using static IP assignment while leveraging cloud-init to install selected packages per machine component and execute various commands to setup an application.

I still say that this shouldn’t be that hard for our customers to setup and hopefully Software Component like I mentioned would save us all from all this complexity, of-course this is beside the fact that you still can do this via various configuration management tools such as Ansible and puppet which by the way vRealize Automation 8 and cloud integrate with today out-of-the-box.

In a high level when the virtual machine first boots up and gets rebooted to be customized due to the dynamic vCenter customization specs that gets created based on the fact we are using the assignment static property ( assignment: static ) within the blueprint code as you see in the screenshot below, I am making sure that during that time, Cloud-init is in a disabled state.

After the customization reboot the virtual machine once, there is a Cron Job that I created on the template that execute at startup after a 90 sec of sleep which is enough time for the virtual machine to be customized, rebooted and connected to the network without running the Cron Job as of yet. After the initial reboot and pass the 90sec mark now the Cron Job execute a shell script that enables cloud-init and initializes it running all the needed cloud-init modules. ( init, Config and Final)

Note: Feel free to increase the 90 sec if you feel you need more time as the virtual machine being customized. 

The End result, the virtual machine is now customized with an updated host-name and an IP from our targeted static IP pool configured for the network its connected to without having to hack the Cloud Config code any further to setup things like the host-name or even configure the network itself, and more importantly without conflicting with cloud-init which what the problem was all along.

Let’s get started, Eh!

• Build a new Ubuntu 16.04 or 18.04 virtual machine from the certified ISO

• Once the virtual machine is up and running update the list of available packages and install any new available version of these packages that you have to update your template

sudo apt-get update && sudo apt-get -y upgrade

• Install Cloud-init for Ubuntu 16.04. Ubuntu 18.04 have cloud-init pre-installed so you can skip this step

sudo apt-get -y install cloud-init

• Configure OVF as your Datasource, then save and exit

sudo dpkg-reconfigure cloud-init

• Enable traditional Guest OS Customization GOSC Script by editing /etc/cloud/cloud.cfg file and adding

disable_vmware_customization: true

• Ensure network configuration is disabled in /etc/cloud/cloud.cfg, by adding or un-hashing the following if it exists:

network:
  config:disabled

If a cloud-init network config is not found and no disable option is specified then cloud-init will default to a fallback behavior which is to use DHCP if you happen to reboot the server.

By specifying the “disabled” option we are telling cloud-init not to try and do anything with the network on each subsequent startup which allows the guest OS to use the config that was originally applied to the machine on first run.

• Set Temp not to clear, by editing /usr/lib/tmpfiles.d/tmp.conf  and adding the prefix # to line 11.

#D /tmp 1777 root root -

• Configure Open-vm-tools to start after dbus.service by editing /lib/systemd/system/open-vm-tools.service file and adding the following under the [Unit] section.

After=dbus.service

• Reduce the raise network interface time to 1 min by editing /etc/systemd/system/network-online.targets.wants/networking.service file and changing: ( This not applicable on Ubuntu 18.04 )

TimeoutStartSec=5min to TimeoutStartSec=1min

• Disable cloud-init on First Boot and until customization is complete by creating this file /etc/cloud/cloud-init.disabled

sudo touch /etc/cloud/cloud-init.disabled

• Create a script your_script.sh in a known location that will be called by a Cron Job that will create later to enable and initialize cloud-init after the customization reboot. The script should contain the following commands:

sudo rm -rf /etc/cloud/cloud-init.disabled
sudo cloud-init init
sudo sleep 20
sudo cloud-init modules --mode config
sleep 20
sudo cloud-init modules --mode final
sudo touch /tmp/cloud-init.complete
crontab -r 

• Configure the script to be an executable

sudo chmod +x your_script.sh

• Create a Cron Job that will run after 90 sec of sleep at boot by typing crontab -e and entering the following:

@reboot ( sleep 90 ; sudo sh /Script_path/your_script.sh )

• Copy the content below for the Template Cleaning script and create your_clean_script.sh. You can replace cloudadmin with your own user that you setup when you installed the Ubuntu OS

#!/bin/bash

# Add usernames to add to /etc/sudoers for passwordless sudo
users=("ubuntu" "cloudadmin")

for user in "${users[@]}"
do
cat /etc/sudoers | grep ^$user
RC=$?
if [ $RC != 0 ]; then
bash -c "echo \"$user ALL=(ALL) NOPASSWD:ALL\" >> /etc/sudoers"
fi
done

#grab Ubuntu Codename
codename="$(lsb_release -c | awk {'print $2}')"


#Stop services for cleanup
service rsyslog stop

#clear audit logs
if [ -f /var/log/audit/audit.log ]; then
cat /dev/null > /var/log/audit/audit.log
fi
if [ -f /var/log/wtmp ]; then
cat /dev/null > /var/log/wtmp
fi
if [ -f /var/log/lastlog ]; then
cat /dev/null > /var/log/lastlog
fi

#cleanup persistent udev rules
if [ -f /etc/udev/rules.d/70-persistent-net.rules ]; then
rm /etc/udev/rules.d/70-persistent-net.rules
fi

#cleanup /tmp directories
rm -rf /tmp/*
rm -rf /var/tmp/*

#cleanup current ssh keys
#rm -f /etc/ssh/ssh_host_*

#cat /dev/null > /etc/hostname

#cleanup apt
apt-get clean

#Clean Machine ID

truncate -s 0 /etc/machine-id
rm /var/lib/dbus/machine-id
ln -s /etc/machine-id /var/lib/dbus/machine-id

#Clean Cloud-init
cloud-init clean --logs --seed

#cleanup shell history
history -w
history -c

• Configure the Template Cleaning script to be an executable as well

sudo chmod +x your_clean_script.sh

• Execute the Template Cleaning Script.

sudo ./Script_path/your_clean_script.sh

• Shutdown the virtual machine and turn it into a template.

Shutdown -h now

Note : Just be aware that the cron job might run if you try to update the template for any reason . So make sure if you do pass 90 sec while doing your change is to re-add the /etc/cloud/cloud-init.disabled file and then re-execute the clean up script again before shutting down the template . if you don’t, cloud-init will execute on first boot and you will get the vm customization but your cloud config code wont be applied

Click To See It All In Action On my YouTube Channel !

I have scripts on github that your welcome to download or fork where you can apply on a base image once its build to prepare it for cloud-init use

There are 4 scripts that you can execute on base CentOs/RHEL or Ubuntu to install cloud-init and configure the image template to work with vSphere customization with DHCP or IP Static assignments

There are two files for each of the linux distro, the ones with a myblog at the end of the file name uses a cron job approach that I used in my blog and the one without, uses a custom runonce service approach that we create instead of using a cron job. Both works but at the end these are two different approaches , your welcome to use which ever one you prefer.

The script will also create both the runonce and clean scripts in the /etc/cloud folder before it runs them at the end before shutting down the VM and then you manually converting it to a template.

Important Note:

Make sure after doing a git clone to Convert Windows-style line endings to Unix-style to remove any carriage return character, otherwise you will get an error like this when you try to execute the script :

“Bash script and /bin/bash^M: bad interpreter: No such file or directory [duplicate]”

Though there are some tools (e.g. dos2unix) available to convert between DOS/Windows (\r\n) and Unix (\n) line endings, you’d sometimes like to solve this rather simple task with tools available on any Linux box you connect to. So, here are an example how to use the sed command to do that quickly:

sed -i -e 's/\r$//' scriptname.sh

Happy Template Building! Please share!

The End Eh!

Resources:

https://ubuntu.com/engage/cloud-init-whitepaper https://debconf17.debconf.org/talks/164/ https://cloudinit.readthedocs.io/en/latest/ https://events.linuxfoundation.org/wp-content/uploads/2017/12/Cloud-init-The-cross-cloud-magic-sauce_Smith_moser.pdf https://www.youtube.com/watch?v=RHVhIWifVqU https://www.youtube.com/watch?v=y8WA1BUlT-Q https://linuxtechlab.com/executing-commands-scripts-at-reboot/ https://blogs.vmware.com/management/2019/02/building-a-cas-ready-ubuntu-template-for-vsphere.html http://kb.vmware.com/s/article/56409 https://kb.vmware.com/s/article/59687 http://kb.vmware.com/s/article/59557 http://kb.vmware.com/s/article/2378666 https://blah.cloud/infrastructure/using-cloud-init-for-vm-templating-on-vsphere/ http://ubuntu.com/blog/cloud-init-v-18-2-cli-subcommands http://lucd.info/2019/12/06/cloud-init-part-1-the-basics/