Debugging 6.5 Million PM2 Restarts: Stabilizing MyZubster Sepolia and I-ECO-01

Leader ●2 ●24 ●155
calendar_today • schedule5 min read

Debugging 6.5 Million PM2 Restarts: Stabilizing MyZubster Sepolia and I-ECO-01

Today I completed a significant infrastructure debugging session across the MyZubster Sepolia E2E environment and the I-ECO-01 / UrbanLab backend.

What initially looked like a few unrelated configuration problems turned into a useful example of why production debugging needs to happen layer by layer.

The work involved Node.js, PM2, MongoDB, Google OAuth 2.0, Cloudflare, Socket.IO, and a Linux VPS.

Here is what happened, what was broken, and how it was fixed.


1. Google OAuth Was Reaching the Backend — But Authentication Still Failed

The first problem appeared in the Google social-login flow.

The authorization request itself looked correct.

Google was receiving the expected client ID and redirect URI:

https://sepolia.myzubster.com/api/auth/social/google/callback

The callback also reached the backend correctly.

Debug instrumentation confirmed:

hasCode: true

hasState: true

hasError: false

So the browser redirect, public route, OAuth state and callback routing were all working.

The failure happened one step later: during the authorization-code exchange.

The backend logged:

401 invalid_client

and Google reported that the provided client secret was invalid.

That was an important distinction.

The problem was not the redirect URI.

It was not Cloudflare.

It was not Express routing.

It was not the OAuth state.

It was not the frontend.

The OAuth client credential itself was wrong.

After correcting the configuration, the backend reached:

OAUTH-CALLBACK-ENTER

followed by:

OAUTH-CALLBACK-SUCCESS

The server-side Google OAuth flow was working.


2. The .env File Was Correct — The Running Process Was Not

The next issue was more subtle.

The .env file contained:

FRONTEND_URL=https://sepolia.myzubster.com

and:

GATEWAY_PUBLIC_URL=https://sepolia.myzubster.com

Yet after successful OAuth authentication, the application could still redirect toward an old Vercel deployment.

This highlighted an important operational detail:

Changing .env does not automatically mean a long-running process is using the new values.

I inspected the configuration at several levels:

  • .env
  • Node + dotenv
  • PM2 process configuration
  • actual process environment
  • OAuth redirect generation
  • backend logs

The active PM2 environment was then aligned with the Sepolia deployment:

FRONTEND_URL=https://sepolia.myzubster.com

PUBLIC_APP_URL=https://sepolia.myzubster.com

GATEWAY_PUBLIC_URL=https://sepolia.myzubster.com

After that, the active process environment was verified directly.


3. Then I Found a Process With More Than 6.5 Million Restarts

While checking the rest of the VPS, another issue appeared.

The UrbanLab / I-ECO-01 PM2 process was shown as online, but its restart counter was enormous:

6,548,384 restarts

That immediately suggested a crash/restart loop.

This is one of the reasons I do not treat:

PM2 status = online

as proof that an application is healthy.

A process can start, crash immediately, restart automatically and still repeatedly appear online when inspected at the wrong moment.


4. Finding the Root Cause

PM2 was executing:

/opt/I-ECO-01/server.js

I inspected the server entrypoint and found these dependencies:

dotenv

express

cors

helmet

mongoose

http

socket.io

winston

Then I compared those requirements against package.json.

The project only declared:

cors

dotenv

express

mongoose

The logs confirmed the immediate failure:

Error: Cannot find module 'helmet'

The server crashed at:

const helmet = require('helmet');

Because PM2 automatically restarted the process, the cycle repeated continuously.


5. Fixing the Dependency Graph

Instead of installing only the first missing package and waiting for the next crash, I inspected all external imports in the server entrypoint.

Three npm dependencies were missing:

helmet

socket.io

winston

http did not require installation because it is part of Node.js itself.

The missing dependencies were installed and persisted to package.json.

The resulting application dependencies now include:

  • Express
  • CORS
  • dotenv
  • Helmet
  • Mongoose
  • Socket.IO
  • Winston

Each new module was then tested independently with require() before restarting the service.

All modules loaded successfully.


6. Application Startup Was Not Enough — I Tested Stability

After restarting UrbanLab, the application successfully initialized its Escrow controller.

The server reported:

Server started on port 5002

MongoDB connected successfully.

The health endpoint:

http://127.0.0.1:5002/health

returned:

status: ok

service: I-ECO-01

mongodb: connected

But I still did not consider the problem solved.

The original issue was a restart loop, so the correct test was to verify process stability over time.

Before the stability test:

PID = 1026452

restarts = 6548384

After 15 seconds:

PID = 1026452

restarts = 6548384

Same PID.

Same restart counter.

The loop had stopped.


7. Escrow Infrastructure Is Running Again

Once the dependency problem was resolved, the I-ECO-01 Escrow controller initialized with:

Threshold: 2/3

Timeout: 24 hours

Fee: 0.5%

The service also initialized its HTTP API and WebSocket infrastructure.

MongoDB remained connected and the health endpoint continued responding successfully.

So the service moved from an uncontrolled crash loop to an operational backend.


8. Old Logs Can Be Misleading

One interesting detail appeared after the repair.

Running:

pm2 logs urbanlab

still displayed:

Cannot find module 'helmet'

At first glance, that could suggest that the problem still existed.

But those entries were historical.

The current process had already successfully loaded Helmet, initialized the application, connected to MongoDB and remained alive with the same PID.

This is why timestamps, process IDs, restart counters and health checks matter when reading operational logs.

A log file can contain a real error that is no longer a current error.


9. What I Verified

By the end of the session:

Google OAuth provider: operational

Google OAuth callback: operational

OAuth authorization-code exchange: successful

OAuth state handling: verified

Sepolia public URL configuration: corrected

PM2 runtime environment: verified

I-ECO-01 Node.js dependencies: repaired

UrbanLab restart loop: stopped

MongoDB: connected

Health endpoint: healthy

Escrow controller: initialized

2-of-3 multisig configuration: loaded

Socket.IO: initialized

PM2/systemd environment: verified


10. The Main Lessons

This debugging session reinforced several useful principles.

online does not mean healthy

A process manager tells you about process state.

An application health endpoint tells you about application state.

You often need both.

.env does not equal runtime environment

Always verify what the running process actually sees.

This is especially important when using PM2, containers, systemd or other long-running process managers.

Debug OAuth one stage at a time

OAuth involves multiple independent stages:

authorization request → provider consent → callback → state verification → authorization-code exchange → profile retrieval → application session → frontend redirect.

Logging each boundary makes failures much easier to isolate.

Fix dependency problems systematically

When one require() fails, inspect the entire entrypoint instead of repeatedly installing dependencies one crash at a time.

Verify the original failure condition

The problem was a restart loop.

Therefore, successful startup alone was not enough.

The actual success condition was:

same PID + unchanged restart counter + healthy endpoint over time.


Final Result

The MyZubster Sepolia environment is now significantly more stable.

Google OAuth can successfully complete its backend authentication flow.

The Sepolia runtime URLs are correctly propagated to the active backend process.

I-ECO-01 is connected to MongoDB.

The Escrow service is initialized.

And, most importantly, a Node.js service that had accumulated more than 6.5 million PM2 restarts is now staying alive.

Sometimes the most useful development work is not adding another feature.

It is understanding exactly why the existing system is failing — and proving that the fix actually removed the failure.

Stack: Node.js · Express · MongoDB · PM2 · Google OAuth 2.0 · Socket.IO · Cloudflare · Linux · Sepolia

NodeJS #DevOps #WebDevelopment #BackendDevelopment #Debugging #PM2 #MongoDB #OAuth2 #GoogleOAuth #SocketIO #Web3 #Sepolia #Cloudflare #Linux #MyZubster #CoderLegion

🔥 Join developers growing publicly
Share your knowledge, build in public, and grow your developer presence with a global community.

More Posts

Cloudflare Blocked My Server, So I Made My Visitors' Browsers Do the Fetching

bryanhamiltondevverified - Nov 15

🚀 MyZubster: Completing a Real Ethereum Sepolia Marketplace E2E Payment Flow

Myzubster - Sep 26

I deleted 44.6 GB and df moved by 5

BashSnippets - Nov 10

I spent years trying to get AI agents to collaborate. Then Opus 4.6 and Codex 5.3 wrote the rules

snapsynapseverified - Apr 20

ERC20 Edge Cases Every Smart Contract Engineer Should Know

BinnaDev - Jun 24
chevron_left
4.4k Points • 181 Badges
Rimini
112Posts
17Comments
33Connections

Related Jobs

View all jobs →

Commenters (This Week)

3 comments
1 comment
1 comment

Contribute meaningful comments to climb the leaderboard and earn badges!