AWS root filesystem recovery

 1. Shutdown the instance which is affected.

2. Take root disk sanpshot.

3. Detach the root disk and attach to rescue server.

4. Fix the issue by mounting  'mount -t xfs -o nouuid /dev/xvdf4 /mnt'

5. Reattach the disk back to affected server. 

RHEL 7 GRUB reinstall

 grub2-install /dev/sda

grub2-mkconfig -o /boot/grub2/grub.cfg

RHEL boot with network - Troubleshoot 'Login error on ssh and console'

Boot to Troubleshooting mode and select 1st option:

chroot /mnt/sysimage 

Run below command.

Login error: tail - f /var/log/secure - Found pam module missing. /usr/lib64/security.

reinstalled pam after connecting to network. But found pam_unix.so gettting deleted. Disabled mcafee and found it's working. 

ip link

 ip link set dev eth0 up

ip addr add 192.168.122.250/24 dev eth0

ip route add default via 192.168.122.1

https://access.redhat.com/solutions/2626631

OS update ansible playbook

 [cinil@patchmanager ansible_playbooks]$ cat osupdate-preparation.yml

---

  - name: Osupdate

    hosts: all

    become: yes

    become_user: root

    tasks:

      - name: taking backup of important files.

        shell: mkdir /var/tmp/security_patching_bkp;cd /var/tmp/security_patching_bkp;df -h>df.txt;ifconfig -a>ifconfig.txt;uname -a>uname.txt;ps -eaf>ps.txt

        register: result

      - name: Clean yum

        shell: yum clean all

        register: yumclean

      - name: yum list security

        shell: yum list-security > /var/tmp/security_patching_bkp/yum_list-security.txt

        register: yumlist_security

      - name: Yum security update

        shell: yum update --security -y

        register: yumsecurityupdate

      - name: Yum security update skip-broken

        shell: yum update --security -y --skip-broken

        register: yumsecurityupdateskipb

      - name: Count the boot files before reboot

        shell: cd /boot/;ls |grep `rpm -q --last kernel | awk 'NR==1{sub(/kernel-/,"");print $1}'`|wc -l

        register: countbootfiles

      - fail:

          msg: "Kernel update failed"

        when: countbootfiles.stdout != "5"

[cinil@patchmanager ansible_playbooks]$

[cinil@patchmanager ansible_playbooks]$ cat Osupdate-Reboot.yml

---

  - name: Os-Security update Server reboot

    hosts: all

    become: yes

    become_user: root

    tasks:

      - name: Count the boot files before reboot

        shell: cd /boot/;ls |grep `rpm -q --last kernel | awk 'NR==1{sub(/kernel-/,"");print $1}'`|wc -l

        register: countbootfiles

      - name: Checking reboot required or not

        shell: LAST_KERNEL=$(rpm -q --last kernel | awk 'NR==1{sub(/kernel-/,""); print $1}'); CURRENT_KERNEL=$(uname -r); if [ $LAST_KERNEL != $CURRENT_KERNEL ]; then echo 'reboot'; else echo 'no'; fi

        ignore_errors: true

        register: reboot_hint

      - name: Rebooting if required...

        shell: ahutdown -r now "Reboot required for updated kernel"

        async: 0

        poll: 0

        when: reboot_hint.stdout.find("reboot") != -1 and countbootfiles.stdout.find("5") != -1

        register: rebooting

      - name: Waiting for thing to reboot

        pause: seconds=180

        when: rebooting|changed

      - fail:

         msg: "Server not rebooted due to Kernel update fail or wrong reboot"

        when: reboot_hint.stdout.find("reboot") == -1 or countbootfiles.stdout.find("5") == -1

[cinil@patchmanager ansible_playbooks]$

Move root disk in LVM

 https://access.redhat.com/solutions/1609793

fdisk /dev/sdc        ---- Create partition similar to root disk also mark boot disk label.

mkfs.xfs /dev/sdc1   --- make boot disk.

mount /dev/sdc1 /mnt

cp -a /boot/* /mnt/

grub2-install --boot-directory=/mnt /dev/sdc

Update /etc/fstab

blkid /dev/sdc1

Reboot the system.

grub2-mkconfig -o /boot/grub2/grub.cfg

Extend VG

vgextend systemvg /dev/sdc2

move LV one by one

pvmove -b -n rootlv /dev/sda2 /dev/sdc2

Monitor the progress

lvs -a -o+devices