At some point the PAAS machine was rebooted and the drives for the middleware instance did not remount which caused the following error when the backups were taking place:
Further investigation, once connected to the server, you will notice the drives have disappeared from the machine:
And notice errors in the /var/log/messages.
Log onto the machine as root and run lsblk and take note of the vg_binaries with no lvm :
# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT xvdb 202:16 0 21G 0 disk ├─xvdb1 202:17 0 500M 0 part /boot ├─xvdb2 202:18 0 4G 0 part [SWAP] └─xvdb3 202:19 0 16G 0 part / xvdc 202:32 0 14G 0 disk ├─vg_binaries-lv_tools (dm-1) 252:1 0 10G 0 lvm /u01/app/oracle/tools ├─vg_binaries-lv_mw (dm-2) 252:2 0 2G 0 lvm └─vg_binaries-lv_jdk (dm-3) 252:3 0 2G 0 lvm xvdd 202:48 0 6G 0 disk └─vg_domains-lv_domains (dm-0) 252:0 0 6G 0 lvm
Mount them manually, once the mount point has been established.
# mount /dev/mapper/vg_domains-lv_domains /u01/data/otd-instance # mount /dev/mapper/vg_binaries-lv_mw /u01/app/oracle/middleware # mount /dev/mapper/vg_binaries-lv_jdk /u01/jdk
Start any services you had on the machine.
So currently if the machine reboots, we have to manually mount the drives, until the RCA has been generated.
If you do wish to start OTD manually as the Oracle user :
$ export JAVA_HOME=/u01/jdk $ /u01/data/otd-instance/admin-server/bin/startserv