vSphere – Networking – ESXi Single NIC VDS Management Migration

Well, I wasn’t sure how to name this blog as VMware continues to use all kinds of different lingos for all of their bells and whistles. I had the unique opportunity to begin working with migrating management interfaces or also know as vmkernel interfaces around from VSS to the DVS switching. This present a lot of struggles but it seems to me that VMware has really improved this functionality in the later versions of vSphere. I recall running into many kinds of issues when doing this on 4.0. So far using a vCenter 5 server with a mix of 4.1 and 5.0 host testing has proved to be seamless and non-interruptive. However, I would still highly recommend considering all your options and testing this method THOROUGHLY before ever touching production environments.

I was able migrate a single physical NIC running ESXi management from a VSS to a VDS. This video covers how I did that. The reason for the video was because I got all kinds of senseless google links when trying to search for something documented. So, I did myself a favor and published one.

Remember, this is a test and this is only applicable for me to use in a few environments. In most cases I use redundant NICs. Now the real kicker about this is that to migrate from a VDS to a VSS requires a bit more thinking and planning. Especially if you only got access to a single PNIC. Maybe I will cover that some other time… for now try to use two. Also, this may be a solution for environments running single 10GB and need to use PVLANS or centralize managment.

Update vSphere 5 – My two cents err problems

What’s the deal man?

Well to be honest I have ran into two very specific issues and what I want to iterate is how crucial it is to review updates before just deploying a normal vSphere 5 implementation. First off, I want to say that in the middle of my experience with performing the upgrade to vSphere 5 the release of Update 1 occurred. So with that being said comes the dilemma. Coordinating the update process and procedure should always be critical. You should also do your due diligence and review the updates along with bugs. I have to honestly give credit to the VMware Community which has definitely allowed me to identify problems before hand and how to avoid and workaround those. Now on to my issues.

Issue Number 1:Broken sVmotion (Storage Migration)

Well, this one was obvious but being the optimist I am didn’t think I would run into this little issue. However it appears to be a ESXi special feature for vSphere 5! I would highly recommend reviewing the following issues if you are having problems performing storage vMotions on vSphere 5/vCloud 1.5. I believe it is actually an issue with the ESXi hypervisor because prior to Update 1 there was a patch you could install on your ESXi box. Please see the following references for resolution:
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2012122

FIX? > Install UPDATE 1 or ESXi Patch 2

Issue Number 2: vCenter Network Alarm Feature!

So, key words to stress in this issue is probably one that makes many CRINGE. Test and Prod should always be the SAME > We all know how important that is but SERIOUSLY how many of us actually MIRROR everything even the alarms? This is more of an issue with standards and procedures then anything… again I am reminded of the 9 parts planning and 1 part implementing or the “Your poor planning doesn’t account for an emergency on my part”.

If the following statement doesn’t tell you what happened then this KB most certainly can..
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2007231‘

FIX? > Yeah just the read the kb its quite ridiculous… oh wait just install update 1?

So, I am writing this tell you that I would recommend applying or using the in place upgrade for vSphere 5 Upate 1. Oh, and just so you know I warned it still doesn’t support the following build number:

NOTE: I would highly recommend to updating from vanilla 4.1 to avoid the special VMware feature of PSOD.

Last but not least I equally thought it would be important to highlight a video that we can all share and relate to when facing unexpected results…. It’s not exactly the same but I can definitely relate to the frustrations..

Enjoy!

vSphere 5 – Storage pt.2 vCloud and Vsphere Migrations

The point..

So on my last post I covered some things to think on when looking at the new VMFS-5 partitions. Obviously the point in moving to the new VMFS would be to gain all the benefits as explained in that previous post. One thing you will see in this post are just the types of migrations. I also want to highlight that I shared some resources on the bottom for those of you who may want to review some deeper highlights. Obviously there isn’t a ton of documentation out there highlighting this nor the special *features* for vSphere 5 (sVmotion issues??) that you may run into. So let hope I do this yet further justice. On to the blog!

Adding VMFS-5 to the vCloud

1. Log in to vSphere and ensure you have a new LUN provisioned (covered above in how to:)
2. Log into vCloud Director Web Interface and you must be an administrator.
3. Click “System” tab and click on Provider VDC. Right click a PVDC and select “Open”
4. After opening the PVDC select the Datastores Tab and then click the +/- button to add/remove datastores

1. Browse through the datastores by clicking the > button or by searching in the top right. When you have located your datastore highlight it and then click the button then click “OK”. Disregard the warning.

(Note: the yellow highlights are ways you can search and browse through datastores. This is very handy when there are many to look through)

(Note: Highlight in yellow shows the datastore added successfully. This is a 20TB Datastore)

You will now see the datastore in the datastore summary tab for that PVDC

Migrating Virtual Machines for vCloud Director to the “new” VMFS-5 LUN.

1. Make sure the vApp is NOT a linked clone. If it is a linked clone defer to the references below.
2. Ensure the Datastore you want to Storage Motion the Virtual Machine to is also provisioned to the Org VDC. Do this by opening the Org vDC and selecting the “Datastores” Tab.

   Note: you can see both datastores are attached to this VDC with the organization known as App1
   

3. You could then log-in to vSphere client with the following noted vCenter and perform a storage vMotion. Another way of doing a Storage vMotion could be by using William Lam’s script he wrote as well. (see references below)
4. If you need to perform the sVmotion defer to the following method below.

NOTE: I would highly recommend that you roll out update 1 to all vCloud components. This addresses a few major fixes that will allow for operations to run more smoothly. More importantly, the only way to sVmotion vCloud VMs is to turn them off. This is a pretty common issue with vanilla vsphere 5/vcloud 1.5 roll outs. I also experienced this problem. For more information please see references at the bottom.

Migrate a Virtual Machine with Storage VMotion in vSphere

Use migration with Storage VMotion to relocate a virtual machine’s configuration file and virtual disks while the virtual machine is powered on. You cannot change the virtual machine’s execution host during a migration with Storage VMotion. (Note: that if VM is managed by vCloud and not at 1.5 update 1 you will need to possibly power off the virtual machine to perform the svmotion. If the virtual machine is a fast provisioned vm (linked clone) then you will need to perform the sVmotion through an API.

Procedure

• Ensure you are not moving vCloud vApp if you are please follow the above process first.
  
• Display the virtual machine you want to migrate in the inventory.
  
• Right-click on the virtual machine, and select Migrate from the pop-up menu.
  
• Select Change datastore and click Next.
  
• Select a resource pool (the same) and click Next.
  
• Select the destination datastore:
  To move the virtual machine configuration files and virtual disks to a single destination, select the datastore and click Next.
  To select individual destinations for the configuration file and each virtual disk, click Advanced. In the Datastore column, select a destination for the configuration file and each virtual disk, and click Next.
  
• Select a disk format and click Next:
  
• Option Description
  Same as Source Use the format of the original virtual disk.
  If you select this option for an RDM disk in either physical or virtual
  compatibility mode, only the mapping file is migrated.Thin provisioned Use the thin format to save storage space. The thin virtual disk uses just as
  much storage space as it needs for its initial operations. When the virtual disk
  requires more space, it can grow in size up to its maximum allocated capacity.
  This option is not available for RDMs in physical compatibility mode. If you
  select this option for a virtual compatibility mode RDM, the RDM is
  converted to a virtual disk. RDMs converted to virtual disks cannot be
  converted back to RDMs.

  Thick Allocate a fixed amount of hard disk space to the virtual disk. The virtual
  disk in the thick format does not change its size and from the beginning
  occupies the entire datastore space provisioned to it.
  This option is not available for RDMs in physical compatibility mode. If you
  select this option for a virtual compatibility mode RDM, the RDM is
  converted to a virtual disk. RDMs converted to virtual disks cannot be
  converted back to RDMs.

  NOTE: Disks are converted from thin to thick format or thick to thin format only when they are copied from one
  datastore to another. If you choose to leave a disk in its original location, the disk format is not converted, regardless of the selection made here.

• Review the page and click Finish.
  
• A task is created that begins the virtual machine migration process.
  

References:

Linked Clones:
http://www.virtuallyghetto.com/2012/04/scripts-to-extract-vcloud-director.html
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1014249

Storage Motion Issue:
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2012122

How To’s sVmotion CLI/VCO style:
http://www.virtuallyghetto.com/2012/02/performing-storage-vmotion-in-vcloud.html
http://www.virtuallyghetto.com/2012/02/performing-storage-vmotion-in-vcloud_19.html
http://geekafterfive.com/2012/03/06/vcloud-powercli-svmotion/
http://geekafterfive.com/tag/vcloud/
http://pubs.vmware.com/vsphere-50/topic/com.vmware.ICbase/PDF/vsphere-esxi-vcenter-server-501-virtual-machine-admin-guide.pdf

Storage Considerations for vCloud:
http://www.vmware.com/files/pdf/techpaper/VMW_10Q3_WP_vCloud_Director_Storage.pdf

vSphere Networking 101 – Renaming VMnics – Backup and Restore

This is a super simple tutorial that I wanted to do on how to rename VMnics. This is great for network card replacement and is just a good thing to know in case something does go south on a hardware replacement. Let’s move on.

SSH to your host and try to keep in mind to use a DRAC or ILO just in case. There should be no need to do this if you are not touching the management or service console uplinks for ESX or ESXi.

1. run:
2. cp -p /etc/vmware/esx.conf /etc/vmware/esx.conf.backup (This will backup the current configuration seek kb here)
3. VI /etc/vmware/esx.conf
4. scroll down and locate the Dev/ids and these will be followed with a =vmnic#
5. Type i
6. go over to the VMnic you wish to modify
7. delete of backspace
8. Type in the right name
9. Hit “ESC”
10. Type “:wq” yes that is a colon wq.

You can either reboot or try cycling services but that is pretty much all that really to it. Enjoy and thinks for stopping by!

Youtube video:

[Resolved] VMware Workstation 8 – Windows XP VM Hang Issue

This video explains how I solved my own issue after I upgraded to VMware Workstation 8. It seems going throught he process of removing and adding virtual devices narrowed mine down the A: drive or aka Floppy Disk. I just simply disconnected it through VMware Workstation and I would recommend removing it entirely if you do not need it. To this you would need to power down the VM. Just for the record I did an upgrade for VMware workstation, however it completely uninstalls and reinstalls the new version.

1. Power Down the VM

2. Right Click and go to “Settings”

3. Click on the Floppy Drive

4. Click Remove

5. Click Ok

*NOTE* You can also just uncheck the connect at power on option as well. I hope this fixes your issue as well.

VMware vSphere 4 and 5 Labs – Foundations – The different kinds…

To Build a lab:

I have been thinking a lot about how there seems to be a few gaps in the VMware community when it comes to learning to set up a VMware vSphere lab environment. So I thought I would take the time to try and put together a full on post dedicated to resources on building a VMware Lab. When I first thought about this I wasn’t sure if I wanted to do a full A-Z build. Covering every single feature or deployment, but often times I would rather not re-invent the wheel. There are MANY post covering how to do this in general but I wanted to make a point of identifying the types of labs that you can set up and how to exactly go about it as well. The key word is “lab” so you don’t want to spend a ton of money (unless you have it) on your lab. To start off there are a multitude of setups you can do and many ways you can do it. I also want to stress that if you are getting ready for your test then YOU need to have one of these labs.

vSphere Lab video 2 Cents and quick overview! (this is my fist video post)

Nested VMware vSphere Lab

1. Hosted on a Desktop Virtualization Product Like VMware Workstation 7 or 8
2. Allows for easy HCL compliance
3. Does require a robust desktop
4. Can get slow depending on what you’re doing (design)
5. Networking is all virtualized (plus)
6. Storage can be virtualized or something like iSCSI can be used
7. Mobility (can move VM’s around between desktops and laptops)

Physical VMware vSphere Lab

1. Runs ESXi as bare metal
2. Is more expensive
3. “Real World” set up so is truly a lab
4. Must meet HCL
5. Will need Physical Networking (Managed networking highly recommended)
6. Takes longer to build out or rebuild
7. Can run nested labs on top of ESXi (pretty much using ESXi in the way you would use VMware Workstation)
8. Storage can be virtualized or something like iSCSI can be used
9. Can move hosted VM’s but the physical systems are not portable/mobile (depends I guess)

In a nutshell I will be covering the nested set-up since that seems to be the less expensive rig. I also love the fact that I can move it around to my laptop and desktop which is quite handy. Also fairly easy to backup as well.

***Disclaimer: The thoughts and views expressed on VirtualNoob.wordpress.com and Chad King in no way reflect the views or thoughts of his employer or any other views of a company. These are his personal opinions which are formed on his own. Also, products improve over time and some things maybe out of date. Please feel free to contact us and request an update and we will be happy to assist. Thanks!~

VMware Certified Professional on vSphere 5! – My Experience

On January 31, 2012 I officially passed my vSphere 5 exam and I am now a VCP 5. The test was very interesting indeed. You hear all sorts of rumors and things of that nature before you take it. I have made a post previously HERE. This pretty much covers everything you need to know. I don’t think I can discuss testing material but what I can tell you is learn a lot about the components of vSphere 5 clustering. Also learn about the Pros and the Cons centered around all the vSphere 5 features.

The test in of itself was actually harder than the vSphere 4 test I took over a year ago. I was also a lot more prepared and had some pressure on me to get it done because I wouldn’t have to pay for training. I would encourage that if any of you are familiar with vSphere 5, Been running it in a lab, and constantly learning new features, know vSphere 4.1 well, then I would go ahead and go take the test. Reason being is that you can end up spending more money waiting to take it later when you can possibly pass it now. Most companies are also going to be more willing to reimburse the expense of an exam then to actually spend money on an course.

Notice my comment on 4.1. Previous VCP were based off of 4.0 only. If you are really familiar with 4.1 then you already have a good lead into the knowledge base of 5.0. If you are a VMware partner then head to the partner portal and take some other course ware related to “What’s New” in vSphere 5. I did pretty good considering the amount of time and the pressure. I would definitely vote on subscribing to the latest VMware Certified Professional on vSphere 5 brown bags over at Cody Bunch’s site.

Either way, now that I have my VCP 5 out of the way I can hopefully attempt to get some advanced certifications around vSphere 4 taken care of.

***Disclaimer: The thoughts and views expressed on VirtualNoob.wordpress.org and Chad King in no way reflect the views or thoughts of his employer or any other views of a company. These are his personal opinions which are formed on his own. Also, products improve over time and some things maybe out of date. Please feel free to contact us and request an update and we will be happy to assist. Thanks!~

Cisco UCS Journey – When to update firmware

Don’t update if its not broke. If it breaks then update it. If you have issues with false alerts you may want to update firmware. I saw this with 1.4j.

The issue is not with the IOM but with the chassis communication bus(i2c bus) and hence the IOM is not getting detected and backplane ports never come up. If you seeing alerts related to PSU and those types of things then you may want to pay attention.

I2C is a bus that provides connectivity between different components in the chassis.

The PCA9541 is an I2C part that helps us control access to the shared devices in a chassis; the chassis serial eeproms, power supplies, and fan modules.

The 9541 I2C mux has known hardware/hang issues that can cause failures to access hardware components on the chassis. This can result in failures to read fan and PSU sensor data (such as fan speed and temp), triggering faults to be raised for the component (such as fan inoperable).

Some early PCA9541s that were used have a bug that if they are switched back and forth between IOM1 access and IOM2 access too quickly, they will get stuck and not allow any connection to the devices behind them.

Action Required:

Required to upgrade firmware version to 1.4(3q) or above.

Workaround to be followed before going for firmware upgrade:

• Reseat all the PSUs one by one in the chassis. Wait for 10min after inserting one unit ,so that it could stabilize.

• Reseat all the Fan Units on the backside of the chassis. Wait for 3min before going for the next one.

• Reseat both the IO modules. Wait for 20min before going for the next one.

• Verifying the i2c counter for the chassis.

• (Requires Down Time)Power cycle to reset all counters to fix issues in the running version.

• (Requires Down Time)Upgrading to firmware version 1.4(3q) or above (2.0 release) for a permanent fix.

Please follow the link to download the 1.4(3q) bundle:

http://www.cisco.com/cisco/software/release.html?mdfid=283612660&release=1.4%283q%29&relind=AVAILABLE&softwareid=283655658&rellifecycle=&reltype=all

Related Issue with firmware version used:

Incorrect behavior of I2C bus or CMC software interpreting I2C transactions?

1. Fans (count 8 or less), PSU (count 4 or less) can be reported as inoperable. State never cleared.
2. Fans are running at 100% rotation rate.
3. UCSM cannot retrieve the PSU/Fan part detailed information
4. Transient errors indicating Fan inoperable, cleared in one minute time interval.
5. LED state does not match faults reported in UCSM and actual health of the system.
6. Incorrectly reported thermal errors on blades and chassis .

Fixes that are promised for 1.4(3q):

1. CSCtl74710 I2C bus access improvements for 9541

PCA9541 (NXP I2C bus multiplexor) workaround to improve bus access for parts built prior of mid 2009. The workaround assures that if internal clock fails to #:start it gets retried. The change designed and works as expected for both PCA9541 and PCA9541A parts from NXP. PCA9541 parts due to the internal clocking bug #:had a high number of bus_lost events.

1. CSCtn87821 Minor I2C driver fixes and instrumentation

New Linux I2C driver has optimization to handle I2C controller and slave devices synchronization. With older driver simple synchronization error could appear as uncorrectable device errors.

1. CSCtl77244 Transient FAN inoperable transition

During UCS (CMC) firmware upgrade and switching to new master/slave mode CMC erroneously takes information from the slave IOM and evaluates fans as inoperable based on stale data.

1. CSCtl43716 9541 device error. Fan Modules reported inoperable, running 100%

Software code routine bug where single bus_lost event followed by successful retry will result in an infinite loop. As result Fans are reported as inoperable and are not controlled by CMC.

1. ??

Removed an artificial cumulative threshold to enable amber color LED upon reaching 1000 bus_lost events. This was implemented as a monitoring mechanism to simplify identification of the PCA9541 devices. This is no longer needed since a proper software workaround is implemented.

Since this email we have started the update to firmware 2.0. This is a separate blog I am going to write because that too was pretty intense. I will provide some additional steps that we performed to lessen the impact. One thing is for certain don’t expect it to NOT be impacting….

***Disclaimer: The thoughts and views expressed on VirtualNoob.wordpress.com  and Chad King in no way reflect the views or thoughts of his employer or any other views of a company. These are his personal opinions which are formed on his own. Also, products improve over time and some things maybe out of date. Please feel free to contact us and request an update and we will be happy to assist. Thanks!~

Network Troubleshooting 101 – vSphere VM Guest (Updated)

General Information (VMware information): (Updated)
What is Beacon Probing?
Well, if you don’t know I would give the following Kb a read.
You can find that here:

Let’s hope I can do this some “Justice”

A few weeks ago I ran into an issue where VM NICS would just randomly go down. The only way I could get them back online was to perform a vMotion of the VM to another host and resetting the port connection on the dvSwitch seemed to fix the issue. So essentially I wanted to highlight some basic troubleshooting steps and things you can do to help you better pin point where the issue may be occurring.

1. From the Guest VM having the issue attempt to ping any other VM on the same host, switch, and portgroup. This will allow to pin point the issue to see if it is related to the physical or virtual networking.
2. Enabling Beacon Probing can help detect upstream failures. (Be sure to read up on it) Enabling beacon probing will increase bandwidth utilization and CPU cycles on an ESXi host so simply consider the tradeoffs.

Why I need to enable it?
VMware recommends to  introduce this change for either permanent or temporary use. Beacon probing as stated above can help or allow us to detect other failures which may occur upstream. When the failure occurs on the VM this will help you isolate the issue as being related to the VMware Virtual networking or the Physical switching (So can troubleshooting). If the failure happens on the VM and the Virtual Switch uplinks trigger an alert this will help you to pin point the issue.

Implementation Instructions:

Enabling Beacon Probing for Distributed Virtual Switch for a vCenter:

1. Before enabling Beacon Probing I will engage James Hendrock to get Bandwidth utilization before and after enabling Beacon probing.
2. Log on into the vSphere Client and connect to any vCenter you want to change it on
3. Browse to Home > Inventory > Networking
4. Expand the following objects in the tree to the left: (see figure)
5. Right Click a Port Group under the dvSwitch switch > Edit settings…
6. Highlight under policies the Teaming and Failover: (see figure)
   

1. Select Network Failover Detection Dropdown (should currently say Link Status only) > Select Beacon Probing (see figure)
   
2. Click Ok
3. Repeat steps 4-7 for all other Port Groups on dvMgmt dvSwitch.

Take Away:

1. Beacon probing is best used with a 3-pNIC configuration with even a N+2 Switching design being highly recommended
2. Beacon probing configured with 2-pNICs means that whether or not you use it in a N+1 configuration it will detect a downstream failure but it has no way of knowing which uplink is bad.
3. In a 2-pnic configuration if one fails it will trigger a redundancy lost message but at the same time just shotgun traffic down both pNICs to ensure communication is sent.
4. If you are going to use Beacon Probing you must also consider the network design.

Resources:
http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=1005577&sliceId=1&docTypeID=DT_KB_1_1&dialogID=273348945&stateId=0 0 273356736
http://blogs.vmware.com/networking/2008/12/using-beaconing-to-detect-link-failures-or-beaconing-demystified.html
http://www.bctechnet.com/?p=10
http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=1012819&sliceId=1&docTypeID=DT_KB_1_1&dialogID=273348945&stateId=0 0 273356736
http://frankdenneman.nl/2011/02/ip-hash-versus-lbt/
http://www.yellow-bricks.com/2010/08/04/high-physical-switch-cpu-load/

***Disclaimer: The thoughts and views expressed on VirtualNoob.wordpress.com  and Chad King in no way reflect the views or thoughts of his employer or any other views of a company. These are his personal opinions which are formed on his own. Also, products improve over time and some things maybe out of date. Please feel free to contact us and request an update and we will be happy to assist. Thanks!~

vCenter Orchestrator Resources and Links

I wanted to only post some resources for VMware vCenter Orchestrator so here they are. I also hear through the grapevine a PowerShell plug-in is coming the best part is that vCloud is also getting the PowerCLI cmdlets as well! Cheers!

vCenter Orchestrator Links and Other Sources:

http://www.vmware.com/support/pubs/orchestrator_pubs.html
http://professionalvmware.com/vcenter-orchestrator/
http://www.vcoteam.info/
http://www.vcoportal.de/
http://downloads.vmware.com/d/details/vco_va_420/dHRAYnRoamViZHAlZA > VCO Appliance

Appliance Download and Automated Installation:
Appliance
Installation – Cody Bunch
Installation 2 – VCO Team
Automated Installation – Lamw

Documentation Download:
Orchestrator Compiled Docs (Admin guides and more)

Other Stuff:
Monitoring SNMP for Ticket Alerting “Use-Case” Plug-In
http://blogs.vmware.com/orchestrator/2011/09/snmp-plug-in-integration-with-vcenter.html
CIM Hardware (Yes, full hardware reporting and automation now..)
http://labs.vmware.com/flings/cim-plugin
Web Based UI Plug-in for VCO:
http://www.vcoportal.de/2011/10/welcome-back-perspectives/
Plug In List (Official)
http://www.vmware.com/products/datacenter-virtualization/vcenter-orchestrator/plugins.html
contains:
vCenter Server
vCloud Director
vCenter Update Manager
CiscoUCS Manager (this is the management application)
MS AD (Microsoft Active Directory)
HTTP-REST (This is also what VCD uses)
SOAP
AMQP protocol/MQ
SNMP

Companies with vCenter orchestrator plug-ins:
RadWare: (Automating Load balancing and auto provisioning of more web servers when load gets high)
http://www.radware.com/Solutions/Enterprise/Virtualization/vDirect.aspx
http://www.youtube.com/watch_popup?v=4rkV3ebQens&vq=medium
http://www.youtube.com/watch_popup?v=6X-J6llsuho&vq=medium
Uptime: (More monitoring)
http://support.uptimesoftware.com/orchestrator.php

***Disclaimer: The thoughts and views expressed on VirtualNoob.wordpress.com  and Chad King in no way reflect the views or thoughts of his employer or any other views of a company. These are his personal opinions which are formed on his own. Also, products improve over time and some things maybe out of date. Please feel free to contact us and request an update and we will be happy to assist. Thanks!~