vCenter 5.5U2 Upgrade on VCSA

So I decided to upgrade my vCenter 5.5U1 VCSA to the latest vCenter 5.5U2 release yesterday and it has been a complete disaster at this point. vCenter continually crashes every few minutes now. I was just poking through logs and found where the service is crashing. I am including the context of the log below.

 

mem> 2014-09-11T19:06:53.383Z [7F02DC3D0700 warning 'Default' opID=HB-host-8637@82541-3778684f] [VdbStatement] SQL execution failed: UPDATE VPX_HOST SET ROOT_RES_CONFIG = ? WHERE ID = ?
mem> 2014-09-11T19:06:53.383Z [7F02DC3D0700 warning 'Default' opID=HB-host-8637@82541-3778684f] [VdbStatement] Execution elapsed time: 2 ms
mem> 2014-09-11T19:06:53.383Z [7F02DC3D0700 warning 'Default' opID=HB-host-8637@82541-3778684f] [VdbStatement] Bind parameters:
mem> 2014-09-11T19:06:53.383Z [7F02DC3D0700 warning 'Default' opID=HB-host-8637@82541-3778684f] [VdbStatement] datatype: 11, size: 1234, arraySize: 0
mem> 2014-09-11T19:06:53.383Z [7F02DC3D0700 warning 'Default' opID=HB-host-8637@82541-3778684f] [VdbStatement] value = "ha-r..."
mem> 2014-09-11T19:06:53.383Z [7F02DC3D0700 warning 'Default' opID=HB-host-8637@82541-3778684f] [VdbStatement] datatype: 7, size: 4, arraySize: 0
mem> 2014-09-11T19:06:53.383Z [7F02DC3D0700 warning 'Default' opID=HB-host-8637@82541-3778684f] [VdbStatement] value = "8637"
mem> 2014-09-11T19:06:53.383Z [7F02DC3D0700 error 'Default' opID=HB-host-8637@82541-3778684f] [VdbStatement] SQLError was thrown: "ODBC error: (00000) - " is returned when executing SQL statement "UPDATE VPX_HOST SET ROOT_RES_CONFIG = ? WHERE ID = ?"
mem> 2014-09-11T19:06:53.417Z [7F02DC3D0700 verbose 'VpxProfiler' opID=HB-host-8637@82541-3778684f] [1-] [VpxdVdb] SaveCLOB: vim.ResourceConfigSpec (took 36 ms)
mem> 2014-09-11T19:06:53.418Z [7F02DC3D0700 error 'commonvpxCommon' opID=HB-host-8637@82541-3778684f] [Vpxd_HandleVmRootError] Received unrecoverable VmRootError. Generating minidump ...
mem> 2014-09-11T19:06:53.418Z [7F02DC3D0700 error 'Default' opID=HB-host-8637@82541-3778684f] An unrecoverable problem has occurred, stopping the VMware VirtualCenter service. Error: Error[VdbODBCError] (-1) "ODBC error: (00000) - " is returned when executing SQL statement "UPDATE VPX_HOST SET ROOT_RES_CONFIG = ? WHERE ID = ?"
mem> 2014-09-11T19:06:53.418Z [7F02DC3D0700 verbose 'commonvpxCommon' opID=HB-host-8637@82541-3778684f] Backtrace: 
mem> --> 
mem> 2014-09-11T19:06:53.487Z [7F02DC3D0700 panic 'Default' opID=HB-host-8637@82541-3778684f] (Log recursion level 2) Unrecoverable VmRootError. Panic!

------ In-memory logs end   --------
2014-09-11T19:06:53.487Z [7F02DC3D0700 panic 'Default' opID=HB-host-8637@82541-3778684f] Section for VMware VirtualCenter, pid=10869, version=5.5.0, build=2001466, option=Release

If anyone has any thoughts let me know.
Enjoy!

6 thoughts on “vCenter 5.5U2 Upgrade on VCSA

    • I ended up rebuilding the vcsa. I was unable to get anything worked out for several days. So I decided to just stay over from scratch.

  1. Been having the exact same issue for 4 days. VMware is working the issue as well and are going to get back with me. I am close to re-building the VCSA from scratch myself.

    • I started having this issue myself and couldn’t find anyone else online with it except for this post, so I thought I would share my solution here for anyone else that lands here via Google.

      Check your postgres log files (/storage/dbvpostgres/pg_log) for an entry that looks like:

      PANIC: checksum mismatch: disk has 0x4addc132, should be 0x7cb3b932
      filename base/16385/24461, BlockNum 37, block specifier 1663/16385/24461/0/37

      This checksum error can be repaired using the embedded postgres tools on the vCenter Appliance. Here are the steps to follow:

      Stop the vCenter services:

      service vmware-vpxd stop

      Stop the postgres service:

      service vmware-vpostgres stop

      Do the repair. Substitute in the block specifier from your log file. It should look like: 1663/16385/24461/0/37

      /opt/vmware/vpostgres/9.0/bin/postgres –single -D /storage/db/vpostgres -c fix_block_checksum=””

      Start postgres and watch the log file to make sure there are no errors:

      /opt/vmware/vpostgres/9.0/bin/pg_ctl start -D /storage/db/vpostgres

      And finally I rebooted the vCenter appliance so it could start the database via its own scripts.

      This was the fix for me, and I hope it’ll help anyone else who lands here!

      The inspiration was taken from VMware’s KB article 2030160 for vFabric (http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2030160).

      • @Daniel – Thanks for the info on this. I actually just had this issue again yesterday. And sure enough, it was a checksum mismatch.

        tm:2015-02-23 02:57:20.379 UTC db:VCDB pid:14903 PANIC: checksum mismatch: disk has 0xbda70eed, should be 0xa4b365bc
        filename base/16385/24612, BlockNum 240, block specifier 1663/16385/24612/0/240
        One issue I ran into was that connecting to the VCSA using root was that I could not run the following:

        /opt/vmware/vpostgres/9.0/bin/postgres –single -D /storage/db/vpostgres -c fix_block_checksum=”1663/16385/24612/0/240″

        It will not let you run this as root. So I had to su to postgres but before you do that you have to edit /etc/passwd for the postgres user and change /bin/false to /bin/bash and then you can do the following.
        su postgres
        /opt/vmware/vpostgres/9.0/bin/postgres –single -D /storage/db/vpostgres -c fix_block_checksum=”1663/16385/24612/0/240″

        And then change /etc/passwd back to /bin/false for postgres once this completed successfully.

        I also rebooted once done and all was good again.

        Thanks a ton for posting this…..

  2. Hello,

    Just wanted to say that the above really helped us out, if I left a tail running on vpxd.log I would get the messages as described in the original post, the Vcenter service would seemingly restart itself automatically though it caused our Veeam backups to fail. Then after an indeterminate amount of time Vcenter would crash completely with the following errors:

    [LDAP Client] Failed to add LDAP entry CN=BF33A949-37CF-453C-ABE6-F62D2048F626,OU=ComponentSpecs,OU=Health,dc=virtualcenter,dc=vmware,dc=int: 0x68 (Already exists)
    2015-07-01T16:47:56.617Z [7F3A106A3740 error ‘linuxvpxLdap_linux’] [LDAP Client] LdapApi::add_s: key (vmw-vc-URL)
    2015-07-01T16:47:56.617Z [7F3A106A3740 error ‘linuxvpxLdap_linux’] [LDAP Client] LdapApi::add_s: key (objectClass)
    2015-07-01T16:47:56.617Z [7F3A106A3740 error ‘linuxvpxLdap_linux’] [LDAP Client] LdapApi::add_s: key (vmw-vc-HealthComponentType)
    2015-07-01T16:47:56.617Z [7F3A106A3740 error ‘linuxvpxLdap_linux’] [LDAP Client] LdapApi::add_s: key (vmw-vc-SSLThumbprint)
    2015-07-01T16:47:56.617Z [7F3A106A3740 error ‘linuxvpxLdap_linux’] [LDAP Client] LdapApi::add_s: key (vmw-vc-HealthComponentName)
    2015-07-01T16:47:56.618Z [7F3A106A3740 error ‘linuxvpxLdap_linux’] [LDAP Client] Failed to add LDAP entry CN=BF33A949-37CF-453C-ABE6-F62D2048F626.vpxd,CN=BF33A949-37CF-453C-ABE6-F62D2048F626,OU=ComponentSpecs,OU=Health,dc=virtualcenter,dc=vmware,dc=int: 0x68 (Already exists)
    2015-07-01T16:47:56.618Z [7F3A106A3740 error ‘linuxvpxLdap_linux’] [LDAP Client] LdapApi::add_s: key (vmw-vc-URL)
    2015-07-01T16:47:56.618Z [7F3A106A3740 error ‘linuxvpxLdap_linux’] [LDAP Client] LdapApi::add_s: key (objectClass)
    2015-07-01T16:47:56.618Z [7F3A106A3740 error ‘linuxvpxLdap_linux’] [LDAP Client] LdapApi::add_s: key (vmw-vc-HealthComponentType)
    2015-07-01T16:47:56.618Z [7F3A106A3740 error ‘linuxvpxLdap_linux’] [LDAP Client] LdapApi::add_s: key (vmw-vc-SSLThumbprint)
    2015-07-01T16:47:56.618Z [7F3A106A3740 error ‘linuxvpxLdap_linux’] [LDAP Client] LdapApi::add_s: key (vmw-vc-HealthComponentName)
    2015-07-01T16:47:56.724Z [7F3A106A3740 info ‘vpxdvpxdMoDiagnosticManager’] [DiagnosticManagerMo] Running support script located at /usr/lib/vmware-vpx/vc-support.sh
    2015-07-01T16:47:56.724Z [7F3A106A3740 info ‘vpxdvpxdMoOptionManager’] [OptionManagerMo] Invoking callbacks for key vpxd.usageStats.persist, pre commit
    2015-07-01T16:47:56.724Z [7F3A106A3740 info ‘vpxdvpxdMoOptionManager’] [OptionManagerMo] Invoking callbacks for key vpxd.usageStats.persist, pre commit
    2015-07-01T16:47:56.724Z [7F3A106A3740 info ‘vpxdvpxdMoOptionManager’] [OptionManagerMo] No change to vpxd.usageStats.persist
    2015-07-01T16:47:56.724Z [7F3A106A3740 info ‘vpxdvpxdMoOptionManager’] [OptionManagerMo] Invoking callbacks for key vpxd.usageStats.level, pre commit
    2015-07-01T16:47:56.724Z [7F3A106A3740 info ‘vpxdvpxdMoOptionManager’] [OptionManagerMo] Invoking callbacks for key vpxd.usageStats.level, pre commit
    2015-07-01T16:47:56.725Z [7F3A106A3740 info ‘vpxdvpxdMoOptionManager’] [OptionManagerMo] No change to vpxd.usageStats.level
    2015-07-01T16:47:56.725Z [7F3A106A3740 info ‘vpxdvpxdMoOptionManager’] [OptionManagerMo] Invoking callbacks for key vpxd.usageStats.duration, pre commit
    2015-07-01T16:47:56.725Z [7F3A106A3740 info ‘vpxdvpxdMoOptionManager’] [OptionManagerMo] Invoking callbacks for key vpxd.usageStats.duration, pre commit
    2015-07-01T16:47:56.725Z [7F3A106A3740 info ‘vpxdvpxdMoOptionManager’] [OptionManagerMo] No change to vpxd.usageStats.duration
    2015-07-01T16:47:56.732Z [7F3A106A3740 info ‘vpxdvpxdMoCbrcManager’] [CbrcManagerMo::Init]
    2015-07-01T16:47:56.733Z [7F3A106A3740 error ‘vpxdvpxdMoReverseProxy’] [VpxdReverseProxy] Failed to create https proxy: Resource is already in use: <acceptor p:0x00007f3a01525f60, h:33, >
    2015-07-01T16:47:56.733Z [7F3A106A3740 error ‘vpxdvpxdMain’] [Init] Init failed: ReverseProxyMo::Init()
    –> Backtrace:
    –> backtrace[00] rip 00007f3a0a608014 Vmacore::System::Stacktrace::CaptureWork(unsigned int)
    –> backtrace[01] rip 00007f3a0a4f2132 Vmacore::System::SystemFactoryImpl::CreateQuickBacktrace(Vmacore::Ref&)
    –> backtrace[02] rip 00007f3a0a4538a5 Vmacore::Throwable::Throwable(std::string const&)
    –> backtrace[03] rip 00007f3a116d449f Vmomi::Fault::SystemError::Exception::Exception(std::string const&)
    –> backtrace[04] rip 00007f3a116c7632 /usr/lib/vmware-vpx/vpxd(+0xea3632) [0x7f3a116c7632]
    –> backtrace[05] rip 00007f3a116d2ec3 /usr/lib/vmware-vpx/vpxd(+0xeaeec3) [0x7f3a116d2ec3]
    –> backtrace[06] rip 00007f3a116b5dae /usr/lib/vmware-vpx/vpxd(+0xe91dae) [0x7f3a116b5dae]
    –> backtrace[07] rip 00007f3a08577c16 /lib64/libc.so.6(__libc_start_main+0xe6) [0x7f3a08577c16]
    –> backtrace[08] rip 00007f3a116b5009 /usr/lib/vmware-vpx/vpxd(+0xe91009) [0x7f3a116b5009]
    –>
    2015-07-01T16:47:56.735Z [7F3A106A3740 warning ‘VpxProfiler’] ServerApp::Init [TotalTime] took 2118 ms
    2015-07-01T16:47:56.735Z [7F3A106A3740 error ‘Default’] Failed to intialize VMware VirtualCenter. Shutting down…
    2015-07-01T16:47:56.735Z [7F3A106A3740 info ‘vpxdvpxdSupportManager’] Wrote uptime information

    (Pasting in in case someone googles a similar issue)

    The LDAP errors I tried to fix using a VMware KB article that pointed to a stale ADAM record, but it hosed the LDAP database when I tried it to fix it and I had to roll back to a snapshot.

    The only KB references I could find to the “failed to create https proxy: Resource is already in use: <acceptor p:0x00007f3a01525f60, h:33, >” error seemed to be for when someone had started IIS on a Windows VCenter server – clearly not applicable here, so both of the above stumped me.

    After much googling I came across this (just before I was about to give up and rebuild from scratch), tried the fix at the weekend (I too had to change to /bin/bash for the postgres user) and touch wood we’ve not had any issues since. It even seemed to fix an issue whereby the VCenter SMTP service wouldn’t start!

    I’ve logged the above with VMWare support and ask them to consider putting it into a KB article, but for now I hope this post will be of some use to someone if they ever have a similar issue. Thank you both for posting your findings too.

Leave a Reply

Your email address will not be published. Required fields are marked *

*