noshornoshor

by System Admin

Odoo downtime support

As we’re bind by an NDA, we can’t share the company name, we will be referring to it as XYZ.

XYZ s company which provide odoo services in Netherland and Brussels , They are hosted their all odoo applications on aws.  They hosted all their  odoo customer on separated instances.  AWS support for XYZ provide by me.

At 3:30 am  CET time zone  one of ec2 instance  goes down, One of major client van praag hosted on  that ec2 instance (named as “van praag”)  , monitoring alert tool  send mail to me and XYZ team about instance is not reachable. XYZ  team was start looking into this issue with me At 3:37 CET  ,  John (Owner of XYZ) told  me that we need to fix that issue before 7:00 am CET, so I had only 4 hours.  I did stop the  van praag ec2 instance and  detached root  ebs volume ( where odoo and db hosted ) from the  van praag ec2 instance , then I did create new ec2 instance (named as  “TEST”)  and attached earlier detached root volume to “TEST” ec2 instance as external volume called /dev/xvdf  and  then I was start evaluating volume , I found that all data were safe in the volume also I realized  that something problem with kernel  so I did update kernel setting and detached ebs volume , I did remount again ebs volume as root to van praag  ec2 instance start the instance  , ec2 instance booted correctly but still I was not able to make ssh connection to the ec2 instance as well instance was not reachable on all port at all , meanwhile  at 4:15 am CET  I did ask to XYZ  team that  they need to  start found latest db backup which we stored on s3 bucket as well as google drvie  and ready to restore db to new ec2 instance I was planning to install new odoo with your custom static frontend code    if I was not able to recover ec2 instance working in full mode  but this things takes 6- 7 hours because after new installation developer also need to fix some settings, I don’t want to go with this option so I again  detached  volume from the vaan praag ec2 instance and reattached to “ TEST” ec2 instance as external volume then I was start checking ebs volume and found lots of  bad fstab entries in /etc/fastab.conf file so I did remove all unnecessary entries ,detached volume from new ec2 instance and reattach to van praag  ec2 instance as boot volume , at 5:05 AM CET  instance booted successfully and around 5:06 AM CET all port from the instance were reachable, so at  5:10 AM CET I did send confirmation message to XYZ team as well as John  that  van praag ec2 instance now fully recovered and Vaan Praag client odoo application now reachable .

What we learned

From this major  outage  I did  immediately install ebs script which take backup of whole ec2 instance volume (EBS) volume backup in form of snapshot at every day so any outage scenario we could restore ec2 instance in 5 min from the snapshot .

Also I did isolated odoo application from root volume to another external volume , so if root volume get corrupted then our application does not effected .

sysop
About sysop
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s.