Nailing down production issues with Zend Server 9

“Costs of fixing bugs after implementation are 10 times higher than construction phase and 100 times larger than design phase”
– Barry W. Boehm: Software Process Management: Lessons Learned from History

Some things never change. I came across the statement above and was surprised to realize it appeared in an article written almost 30 years ago. Is this still relevant today?

Having developed numerous web applications since I started, back in the year 2000, I couldn’t agree more and this blog covers how Zend Server 9 is here to help.

But first, I’d like to share my experience which is exactly about that. The ‘costs’ mentioned can be translated to business concepts like total cost of ownership (TCO) but in my own small domain it had a completely different interpretation. I used to be terrified by bugs in production (notoriously known as: ‘escalations’) and couldn’t count the times I cursed my job, while staying at the office long nights or even worse, weekends, trying to nail down and solve issues.

Frankly, in most cases I couldn’t solve the problems and had to buy some time and find salvation with a temporary solution (notoriously known as: ‘a workaround’.) At times, things went so ¬bad that my team members and I had to take 24 hour shifts, staying alert. And guess what? Murphy’s Law – it always happened on MY shift…

The challenge – Nailing down production issues as quickly as possible

There were always two main questions that my team and I asked:

1. How can we improve the quality and performance of our app before it reaches production?

Even if we answered this question the best we could, bugs ultimately would find their way into production. They always do. So:

2. How do we nail down production issues and quickly see where problems are occurring?

Sadly, the second question is very problematic: we used to spend hours literally digging in log files (if we had access to them) and even if we found something, it was usually just a fraction of a hint. Once we had that in place, we struggled to reproduce it in QA/staging environments, as our development environment rarely matched production and in many cases didn’t suffice.

black_boxAt the time, we wished we had something like a black box flight recorder. Such a technology could have allowed us to simply trace back on all the code that was executed per a specific request, including variables, return values, and exceptions. It should have been just a step away from a full debugger, which obviously must never be turned on in production, thus was a No-Go.

Clearly, the black box recorder has some footprint on servers and it has to be minimal and configurable. By that I mean, it shouldn’t record each and every request trace but rather turned on when something happens such as an error or event. Otherwise it would have been just as useless as a debugger in production.

Hmmm… in that case, a challenge: such a tool must be super smart. How could it selectively record a request before knowing that the event would occur? By guessing? By that I mean, say the recorder is turned off and alert, as it should be in production. An event occurred at time T, and the recorder is triggered, which is great, but how does it record function calls at T-1? If it doesn’t, then the point is missed. Indeed this is a challenge.

The solution – Code tracing with Zend Server

In Zend Server, we call this black box recorder, code tracing. And we are excited to say that code tracing has been completely remastered in our latest release of Zend Server 9.

Honestly, this feature isn’t new and has been part of Zend Server for several years now, but the capabilities didn’t match what developers were looking for. It was:

• Built with a Flash-based GUI (which is very 90’s and now practically dead)
• Heavy and slow due to the way files were stored and loaded.

Here is what it looked like in Zend Server 8 and below:

old_code_trace

Though not pretty, or a good performer, the old code tracing was really insightful and very detailed. You could configure it and tie it to specific events/URLs in production, to minimize its footprint. The code tracing tree tab displayed the call tree and the statistics per function tab provided a statistical perspective of the data captured in the request. So you could not only trace the execution flow, but also diagnose and profile your code, and determine time spent per function and memory usage. To make things even sweeter, you could manually, ad-hoc, create and run a code trace per URL.

New and improved code tracing with Zend Server 9

First off, we retained the cool functionality and flexibility. Then we completely remastered code tracing with support for PHP 7 and a cleaner, more modern UI (HTML5), coupled with radically improved loading times thanks to on-demand fetching. Configuring code tracing has also become a lot simpler, with new profiles that allow easy switching between common settings, as well as a new development mode allowing hassle-free setup when fine-grained access control lists (ACL) aren’t needed.

Here is how it looks like now:

Zend Server 9 code tracing

So it not only looks nicer, but is significantly faster and efficient.

This is exactly the kind of a black box recorder my team and I could really have used. It would have saved us a lot of time and sweat.

Summary

… hold on, before you’re heading off, I owe you something: remember the challenge I mentioned earlier (scroll up a little)? Well here is how the magic is performed by Zend Server code tracing, in a nutshell:
If you configured Zend Server to trigger on event X, once the event is caught by Zend Server monitoring, the code tracing is turned on for 120 seconds (configurable), recording all requests for this URL before turning itself off. What if the code trace was not created in this timeframe? You have a backdoor and can always access the Zend Server UI to create it manually.

Cool!

And what about my team’s other question (how can we minimize bugs before production)? Stay tuned for my next post.

The following two tabs change content below.

    Mickey Hoter

    Mickey Hoter is a senior product manager at Zend, a Rogue Wave company. He has 15 years of experience in building web-based products - as a developer, team lead, group manager and product manager. Mickey holds a B.Sc. in Information Systems and an MBA, both from the Technion –the Israeli Institute of Technology. He is passionate about productivity, usability and digital experience, which is why he chose to work for Zend. Outside of the office, Mickey enjoys spending his time with his four children and puts a lot of effort in educating them to love Mother Nature, where most of his hobbies take place.

    About Mickey Hoter

    Mickey Hoter is a senior product manager at Zend, a Rogue Wave company. He has 15 years of experience in building web-based products - as a developer, team lead, group manager and product manager. Mickey holds a B.Sc. in Information Systems and an MBA, both from the Technion –the Israeli Institute of Technology. He is passionate about productivity, usability and digital experience, which is why he chose to work for Zend. Outside of the office, Mickey enjoys spending his time with his four children and puts a lot of effort in educating them to love Mother Nature, where most of his hobbies take place.