There are cases where the API server will crash on EC2 without being handled by the JavaScript try/catch. It is possible to run out of resources such as disk or memory on the EC2 server, among a few culprits that are not coding errors. Below are some hints on how to find the issue.
Case: 502 Bad Gateway On Develop Server, App Does Not Fail Locally
Login to the dev server with an SSH connection.
nginx error logs
Monitoring NGINX error log.
tail -f /var/log/nginx/error.log
Interact with the app (in this case viewing statements/archived inventory on the dev server).
Here the app is crashing when executing the request.
2022/06/08 14:14:44 [error] 989#989: *13790 upstream prematurely closed connection while reading response header from upstream, client: 64.139.69.81, server: api-ds-dev.researchblocks.app, request: “GET /v1/inventory/get/?current=1&pageSize=10&total=10&field=id&order=ascend&columnKey=id&column=%7B%7D&filter=%7B%22unique_id%22:[],%22cat_num%22:[],%22name%22:[]%7D&status=2&statusText=Archived HTTP/1.1″, upstream: “http://127.0.0.1:9000/v1/inventory/get/?current=1&pageSize=10&total=10&field=id&order=ascend&columnKey=id&column=%7B%7D&filter=%7B%22unique_id%22:[],%22cat_num%22:[],%22name%22:[]%7D&status=2&statusText=Archived”, host: “api-ds-dev.researchblocks.app”, referrer: “https://develop.omniblocks.app/”
last entry when monitoring the nginx error log.
nginx configuration
The “upstream” for nginx is the application configured to take over URL requests. This can be found in the nginx config files. On our API server that is in /etc/nginx/sites-enabled/default. This is the section of the file that is handling requests starting with a / relative URL, which applies to our request above.
location / {
proxy_pass http://localhost:9000;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}netstat tcp/udp listener investigation
Run netstat to find out what is running on port 9000…
sudo netstat -tulnp
This shows PM2 is running on port 9000. PM2 is part of the API server’s scripts that run at startup. Jenkins starts this for us after doing a build.
Viewing system processes
Show a list of all running processes and their parent/child relationship.
ps -ef –forest
The part we are interested in is down at the bottom…
Why PM2 is running as ubuntu AND Jenkins… that is not good for CPU overhead. Jenkins is the process that runs during a default build. There is a chance the PM2 from Ubuntu is fighting with the PM2 from Jenkins if they are both listening on port 9000.
PM2 management
PM2 User Docs are at https://pm2.keymetrics.io/
Stop the PM2 processes under ubuntu…
whoami
pm2 stop all
pm2 list
