"Curiosity is the very basis of education and if you tell me that curiosity killed the cat, I say only the cat died nobly." - Arnold Edinborough

Martin Fjordvald

Debugging Nginx Errors

Dealing with errors in nginx can be a frustrating experience if nginx isn’t configured correctly. Sadly, the default value for error log is less than optimal and some of the tricks to getting information from nginx are not obvious. This post is intended to be a reference for the tools nginx provide and how to configure them; as well as a general guide on what’s important when facing issues in nginx.

I will probably be vilified for putting this as the first thing, however. Knowing the basics is the most important part to understanding the more difficult issues not just related to syntax. Don’t skip basic nginx syntax or how an nginx request flows. Once you understand these concepts the non-error issues become far more tangible to work with.

The Error Log

For issues where there’s an error involved, having nginx configured correctly is absolutely essential. The error_log directive should be configured with an error log level of warn, either at server or http level depending on whether you want per-vhost logs or server wide logs.

error_log /var/log/nginx/file.log warn;

Anything other than this will just waste your time, either by giving too much information or not enough. With data in our error log lets analyse one of the potential errors that might show up. For this example I will use the often encountered “Unable to open primary script” error.

2013/06/03 12:56:12 [error] 2641#0: *20499586 FastCGI sent in stderr: "Unable to open primary script: /home/user/public_html/wp-login.php (No such file or directory)" while reading response header from upstream, client: 10.0.0.1, server: www.example.com, request: "GET //wp-login.php HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "www.example.com"

If the error looks daunting then not to worry, most of it can be mentally discarded after just a brief glance. Essentially, all the important information here can be distilled down to this.

2013/06/03 FastCGI sent "Unable to open primary script: /home/user/public_html/wp-login.php" request: "GET //wp-login.php"

What’s important here is first the date to know if it’s a recent error. If you’re monitoring your error log and get notice on new entries this can even be discarded. 2nd is the source of the error, our FastCGI backend in this case. The 3rd bit is the actual error and the 4th final bit is the request that caused it. These are all the essentials we need to.

  1. Know where the error came from.
  2. Know where we can most likely reproduce it.
  3. Which words to Google for.

Getting Errors from the Backend

If your backend outputs error messages to stderr then nginx will put these in the nginx error log as well. In the case of PHP this can be done by configuring the php.ini with log_errors set to stderr or on and having no error_log defined. Finally, you also have to set catch_workers_output = yes in the php-fpm.conf file.

For other backends the procedure might be different but the concept the same, make sure errors are output to stderr and nginx will log them.

Getting Debug Output

Despite experience, sometimes understanding what nginx is doing can be quite difficult. Turning on debug mode is usually quite an adventure and often not needed at all with a few handy tricks. Often, all you need is to know the value of a variable to nudge you in the right direction, for those situations we have the return directive.

What this directive allows us to do is output arbitrary text in any server or location block — and it supports variables. This is useful if you want to know if your request is flowing into a location like you expect it to or if you need to know the value of some variable. For instance, to understand what URI you’re asking PHP to execute you could do:

return 200 $request_filename;

It is also possible to output multiple variables in the same return directive.

return 200 $document_root-$fastcgi_script_name;

Finally, you can specify a custom log_format for the access_log, which can be useful if you can’t output debug info to the browser. I find this more cumbersome and usually prefer to have a proper testing environment, but it can do in a pinch.

My Book

If you liked the style of this blog post then check out my nginx book – Instant Nginx Starter. It introduces you to the syntax and flow of nginx and minimizes your time spent debugging errors.

flattr this!

My New Nginx Book: Instant Nginx Starter

During the last few months I have been working on an nginx book for Packt Publishing. The book is called Instant Nginx Starter and is now published!

My goal with this book was to provide a concise introduction to the nginx configuration in a way that allowed people to build on top of the knowledge I provided. This meshed well with the Instant line of books from Packt as they are intended to be short and get the reader started quickly.

The knowledge contained in this book is basically a structured, revised and expanded upon version of the information on this blog. It provides a red-thread to help you process the configuration format and syntax in a logical way. I’m really satisfied with how this book turned out and I hope it will help some of you.

Check it out on Packt Publishing

flattr this!

fastcgi_params Versus fastcgi.conf – Nginx Config History

The nginx source install (and by extension package managers) includes two FastCGI configuration files, fastcgi_params and fastcgi.conf that differ only a tiny bit. To this day they still cause confusion amongst new users due to package managers.

The difference between the two files in the source install is the simple line of: 

fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;

The difference between the two files in most distributions package repositories is nothing, they essentially modified fastcgi_params to match fastcgi.conf.

What this line does is tell PHP which file it should execute, without this nginx and PHP cannot work together. This sounds like a good line to include in the shipped FastCGI configuration file and indeed Igor Sysoev thought so as well. However, due to the configurations of the time this wasn’t as easy as simply adding it in.

Back in the days of 0.6.x when I started using nginx and a few years before this change happened a typical configuration example would look like this.

location ~ \.php$ {
    include fastcgi_params;
    fastcgi_param SCRIPT_FILENAME /var/www/foo$fastcgi_script_name;
    fastcgi_pass backend;
}

Due to community documentation efforts on the wiki people slowly started using the $document_root variable instead of hard coding the root path, however, many people were still using the above configuration many years later.

Because of how array directives inherit and interact the people using the old configuration style made it impossible to include the line in fastcgi_params. Doing this would have meant that SCRIPT_FILENAME would be defined twice and both would be sent to the backend, potentially causing confusing behaviour.

In 0.8.30 (released: 15th of December 2009) Igor then included fastcgi.conf which was the same as fastcgi_params except including the improved SCRIPT_FILENAME fastcgi_param. This meant that the community could now start recommending people include fastcgi.conf instead of recommending moving SCRIPT_FILENAME into fastcgi_params. New articles on the wiki mostly used this, the popular articles were slowly changed to use it and we were promoting it in the IRC support channel.

Of course, an issue back then was that package managers gave nginx very little love and were many versions behind, usually something like 0.6.x versus 0.8.x. The fastcgi.conf file was not included for these people. When they eventually did update they shipped with a fastcgi.conf and a modified fastcgi_params leaving us with a situation where the source install actually differed from the repository install in a non-significant way. While not often, this does still cause the occasional confusing in the IRC channel.

As an aside, I actually prefer 

fastcgi_param SCRIPT_FILENAME $request_filename;

as it takes the alias directive into account, fastcgi_new.conf anyone?

flattr this!

WebSockets in Nginx

Version 1.3.13 of nginx is out and with it comes support for Connection: upgrade and Upgrade header, meaning proxying of WebSockets! Many people have been waiting for this and “are websockets in nginx supported?” is one of the most frequent questions in #nginx on freenode. With that out of the way, time to have a look at the nginx WebSocket implementation.

The documentation provided states the configuration as follows:

location /chat/ {
    proxy_pass ​http://backend;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
}

This is indeed fairly simple and configuring the HTTP version used isn’t even new at all. We can go ahead and improve the versatility a bit by creating a connection variable thus allowing us to move these proxy_set_headers into a generic included file.

map $http_connection $upgrade_requested { 
    default upgrade;
    ''      close;
}

This makes the variable $upgrade_requested available to proxy_set_header Connection and if connection upgrade wasn’t requested then it will default back to “” to not interfere with normal requests. The idea here being that if you always proxy HTTP/1.1 then you don’t need a location specifically to handle WebSockets.

Connection upgrading seems unlikely to be back ported into the stable branch so if you wish to use this you will have to use the development branch. Thankfully development in nginx does not mean that it’s not run-time stable, it just means that the API might change and as such it only affects module authors. Don’t be afraid to install the development version to play around with this new feature.

So far a few limitations are known.

Read More

flattr this!

Understanding the Nginx Configuration Inheritance Model

To understand the inheritance model of nginx you first need to know that nginx operates with multiple blocks of configuration. In nginx such a block is referred to as a context, for instance, a configuration directive placed in server context resides within a server { } block just like a directive placed in http context resides in the http { } block.

There are 6 possible contexts in nginx, here in top to bottom order:

  • Global.
  • Http.
  • Server.
  • If.
  • Location.
    • Nested Location.
    • If in location.
    • limit_except.

The default inheritance model is that directives inherit downwards only. Never sideways and definitely never up. This includes scenarios where you rewrite a request internally from one location to another – every directive in the first location is forgotten and only the second location directives apply for the location context.

Read More

flattr this!

Nginx Taking Funding: A Deal with the Devil?

Yesterday Nginx Inc announced that it had taken $3 million USD in funding. No one deserves this more than Igor Sysoev and it’s hard to believe that Nginx wasn’t commercialized sooner. Well deserved or not, though, whether this funding is good for Nginx or not is up for debate.

To understand the whole aspect of the deal I’ll first cover the worst-case scenario that people might fear happening. I’ll later on cover why this case is unlikely, so please do finish reading before considering me a moron.

The FUD Aspect

Getting funded means a business person has seen potential and decided to invest money to get a return. There’s really no way to deny this, philanthropy simply does not happen in the start-up world unless you’re being funded by your rich but slightly senile aunt. Eventually this business man will want to get a return on his investment and this means the Nginx Inc will have to become profitable. How does an open source project become profitable, though?

  • Going closed source and commercializing the product.
  • Creating a closed source enterprise version to develop alongside open source version.
  • Keeping the core product open and developing commercial extensions of that product.
  • Keeping product open sourced and selling support, training and resources.

Read More

flattr this!