# Postmortem: Internal Server Error




> A thinker sees his own actions as experiments and questions — as attempts to find out something. Success and failure are for him answers above all.    ***Friedrich Nietzsche***



### Overview:

1. Issue Summary
2. Root cause and resolution
3. Timeline
4. Corrective and preventative measures


___________________________________________________________________________


### Issue Summary

After months of development, our web solution is ready, and thanks god our customer was satisfied with his new product, it’s a pizza and fast food delivery web application. our client is very popular in the fast-food delivery market and especially pizza. Although this article is not an advertisement I want to bring you closer to the context of the following events.

After a few months of use, my team and I were called to add new features recommended by our client. After some weeks of hard work finally, it seems all things works fine and it was time to host the update in the web server, a few tests were able to occur a 500 error:


```
HTTP/1.0 500 Internal Server Error
```

I remember it was a rainy weekend evening and that’s the perfect time where the web app receives a massive amount of traffic. From 18:02 pm till 18:38 pm users couldn’t reach the server.


### Root cause and resolution

6:02 pm: we confirmed the outage from different devices.

6:07 pm: trying to curl the IP address and search closely.

6:25 pm: we found out the bug which caused the error.

6:38 pm: the error was fixed.

It was about 36 mn of a blackout.

what follows from this blog is the process we followed to fix this issue, and we learned from it.


### Root cause and resolution

The root cause that occurs that bug it was a ridiculous typo in the extension of PHP files, .phpp instead of .php thing which makes it the request of the web server IP was denied.

It was helpful to use the local “curl” command to get the response from the webserver or in the same such case:

```
root@e701fb5d6808:# curl -sI 127.0.0.1
HTTP/1.0 500 Internal Server Error
Date: Fri, 24 Mar 2017 07:32:16 GMT
Server: Apache/2.4.7 (Ubuntu)
X-Powered-By: PHP/5.5.9-1ubuntu4.21
Connection: close
Content-Type: text/html
```

Another command was helpful in this case is the “ [strace](https://strace.io/) ” command, which is a diagnostic, debugging, and instructional userspace utility for Linux.


### Corrective and preventative measures

First, we had to list all processes that ran on the system using the command “**ps auxf**”:


![1*M2bwVKyx3sBJ-diuTTdzng.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1604761052119/RrYGHlHLW.png)


According to the error we deducted that we must strace the apache PID in one terminal and retried to curl in another one :

```
root@f5c5fc4e021c:~#strace -p 87
Process 87 attached
accept8(8,
....
```

And we found out a strange extension when we used curl again:

    ....
    lstat("/var/www/html/wp-includes/class-wp-locale.phpp", 0x7ffc0a9f7110) = -1 ENOENT (No such file or directory)
    lstat("/var/www/html/wp-includes/class-wp-locale.phpp", 0x7ffc0a9f6fe0) = -1 ENOENT (No such file or directory)
    lstat("/var/www/html/wp-includes/class-wp-locale.phpp", 0x7ffc0a9f9210) = -1 ENOENT (No such file or directory)
    open("/var/www/html/wp-includes/class-wp-locale.phpp", O_RDONLY) = -1  (No ENOENT such file or directory)
    ....


*Bingo !!!*

Under the hood: Our web application is a  [WordPress](https://wordpress.com/alp/?aff=58022&cid=8348279) website running on a LAMP stack, which means that is usually run on (Linux, Apache, MySQL, and PHP), which is a very widely used set of tools.

PHP files use the extension .php and clearly .phpp was a typo occurred the time of update and save, so each time we curl (or a user click on the application link) the web app IP couldn’t recognize the back end files with the extension .phpp, and there it could access the webserver no more.

The lesson that we learned from this incident that if we need to create new files that support language extensions like .php, .html, .py, or whatever, the best practice is to copy past from existing filenames to avoid such crash.


.  .  .


**Resources:**

[How to write an Incident Report/Postmortem](https://sysadmincasts.com/episodes/20-how-to-write-an-incident-report-postmortem)

[How to run a Postmortem](https://blog.serverdensity.com/how-to-write-a-postmortem/)