Posts Tagged 'Linux'

It's a UNIX system, I know this!

I’ve spun up an IRIX machine at home to replace my long dead Linux based file server / general network management box. (It’s a 2-node IP45 Origin rack, so 8 sockets of R14000 lovin’) I don’t really like to have more than 1 active server and 1 active desktop at home at a time for simplicity / heat / space reasons. I’ve been an avid IRIX user since 1990, in the golden days of the Personal IRIS and 4D/480. It was my first UNIX and will always have a special place in my heart. SGI has retired the platform and it will go End of Support in 2013, and the final generation of IRIX hardware is becoming outdated on a GFLOP/watt basis now, and thusly affordable!

IRIX is POSIX, POSIX2 compliant, really strictly. It was based on SysV and later got several BSD extensions added, but anything that isn’t mandated with POSIX probably isn’t there unless they wanted it within the halls of SGI. It’s profiling support, native debugging tools and general sysadmin usability are still unmatched. I can still install and configure an SGI box in my sleep, I will definitely be sad the day the final IRIX hardware is thoroughly useless performance wise.

Enough of the background though, I didn’t want to turn this into a sappy rant over a dead platform. I’ve been porting stuff to IRIX for a decade at least, I find as an exercise it’s very enjoyable. I’ve often found myself porting stuff over only to never use it or even nuke it for the sake of the exercise itself. I find it generally makes you a more ‘aware’ C (and UNIX) programmer, especially of stuff outside of the realm of Linux and GCC. I ran into a really cool one late last night. I’ve been working with a bunch of caching engines, on all layers of the equation (object, code, front end), and while I do all of my actual performance testing in Linux land, I’ve been doing some IRIX ports for shiggles. I ran into in a pretty basic mmap which is what spurred me to write this article:

return mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);

Pretty basic, MAP_PRIVATE specifies that modifications to our memory map are private, that is our changes won’t make it to the file descriptor, it’s copy on write. MAP_ANONYMOUS (at least to me) always seemed like a given too, in Linux/BSD/Solaris it ignores the file descriptors and just gives you a zero’d memory. OS’ all seem to implement this differently, just reading the OS X man page it looks like you can pass flags to the Mach VM about tags and its purgability. IRIX (and HP-UX it seems) completely lacks it. No biggie! Sure enough we can accomplish the same thing thusly!

static int devZero;
devZero = open(“/dev/zero”, 2);
return mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_PRIVATE, devZero, 0);

Easy peasy. Open a file descriptor to /dev/zero, kernel then knows what to do. Worked like a charm, fast in memory caching was a go!

Ok so this is an OS X portability issue, but IRIX lacks daemon() too. The error was just too good to not include in a portability article :P

Let’s talk about some other Linux/IRIX/UNIX/whateverisms. I’m just going to skim the surface but here we go….

You also run into lots of cases where IRIX’s libc does not support many ‘givens’ in the Linux (and even FreeBSD and even sometimes Solaris’) libc. I was working on porting varnish to IRIX on Sunday morning, and while we’re still in the syscall tracing phase due to some misbehaving of its internal code compiler, but I’ll use it as an example as it had two of the most common functions that are easy substitutions I see all the time:

setenv() setenv was added to UNIX version 7 back in 1979, somehow IRIX completely lacks it. It’s pretty easy to throw putenv in for most cases though. Looking at the varnish source we swap:

        AZ(setenv("TZ", "UTC", 1));

for

        AZ(putenv("TZ=UTC"));

Same goes for wait4(), it’s part of System 4 but not specified by POSIX so IRIX completely lacks it. Thankfully it’s what waitpid() is implemented with on many platforms (BSD), so if the *rusage isn’t specified, we can do a direct swap thusly:

        r = wait4(v->pid, &status, 0, NULL);

becomes

        r = waitpid(v->pid, &status, 0);

Varnish needed a few other tweaks, stuff like IOV_MAX isn’t specified in any IRIX headers, but can be gleaned from sysconfig (1024). CLOCK_MONOTONIC is not defined, but thankfully they had Solaris conditionals around it that became Solaris and IRIX conditionals.  Fun.

Onto GCCisms. GCC is a great compiler suite. Supports a ton of software/hardware, but it will never been the fastest on any platform. On SGI’s we have MIPSpro and on PC’s you have Intel’s really awesome compiler suite. (We used to use Intel’s suite at Cedara for our x86 medical imaging software, strictly based on its killer performance). I tend to try and port stuff to MIPSpro unless you run into too many bad GCCisms. Sometimes I think that GCC being the defacto ‘nix compiler, has lead people (likely CS students) to believe thats just “the way things are”.

Zero length arrays i.e.

char contents[0];

Ok so it is a clever way to throw a place holder into a struct, but this could be totally handled with a char *contents; pointer too.. Not portable though…

Variable length arrays i.e.

char str[strlen (s1) + strlen (s2) + 1];

Again, its a clever way to do really basic memory handling without thinking about it, but ugh. C99 does support variable length arrays, but GCC allows them in C89 and C++ code too, just to muck shit up. Don’t use these outside of C99 code, it will not be portable.

There are so many more, but these are the most common I tend to see. GNU has a guide on their extensions to the C language, and totally there is lots of really useful stuff there, but just know that it breaks portability. This rant would have a lot more meaning if people used much other than GCC, but nowadays even embedded development and video game consoles are moving toward GCC. At this point in time I honestly don’t see another compiler suite beating GCC but who knows…

Code, port, hack. Expect more cooking stuff soon :P

So last night we  were debugging some odd Apache behaviour when we discovered that RHEL6 has modified the default number of processes (which includes threads in linux, but I’ll refer to it as procs for the purpose of this article). In RHEL5 (and prior), each user is given the default fork init number of threads (which we’ll discuss shortly). It’s a huge number usually, 106, 496 threads on my RHEL5 dev box here. RHEL6 on the other hand, drops a file into /etc/security/limits.d that limits all users to 1024 procs.

Now, /etc/security/limits.conf and /etc/security/limits.d are read by PAM’s pam_limits.so, so only things that use PAM will ever touch these. Apache starting on a server reboot is not affected by these limits, so Apache will have access to its full 106,496 threads. But if root restarts apache from a shell (/etc/init.d/httpd restart or service httpd restart), Apache will inherit root’s proc limit of 1024! This can be verified with a <?php passthru(‘ulimit -u’); ?> via mod_php.

1024 threads is totally not enough for reasonable operation in Apache. It will begin to kill children off as they seteid() to its designated user as the call fails with errno 11. (seteid() has not always performed an nproc check, this was added in some 2.6 kernel to prevent people from sneaking over the limit) You can work around this by dropping a ulimit -u <much higher number> into /etc/sysconfig/httpd, but my preferred solution is to remove (blank) RHEL6′s new /etc/security/limits.d/90-nproc.conf . (Or I guess you could just never restart your apache, best to reboot the system if theres a problem :P) Thanks Redhat.

Back to that large 106,496 number. Thats the maximum number of procs that nick as a user can spawn, even using ulimit -u he can’t go higher than that. The system’s maximum number of threads (visible in /proc/sys/kernel/threads-max) is 212992 (exactly double) is decided at boot time by the kernel. I decided to do some digging in the kernel source and the maximum number of threads is calculated thusly:

max_threads = totalram_pages / (8 * THREAD_SIZE / PAGE_SIZE);

(defined in kernel/fork.c, called by init/main.c)

PAGE_SIZE is architecture specific, for x86 it’s 4kb (and 4MB but thats a discussion for another day), and is calculated via:

#define __AC(X,Y)       (X##Y)
#define _AC(X,Y)        __AC(X,Y)
#define PAGE_SHIFT      12
#define PAGE_SIZE       (_AC(1,UL) << PAGE_SHIFT)

PAGE_SHIFT is a constant for x86, other architectures support alternate (larger) page sizes. THREAD_SIZE is also architecture specific:

#define THREAD_ORDER    1
#define THREAD_SIZE  (PAGE_SIZE << THREAD_ORDER)

THREAD_ORDER is 1 in x86, but can vary depending on the arch. These result in a PAGE_SIZE of 4096 and a THREAD_SIZE of 8192 for x86. Back to the formula  above, we can fill it in as:

max_threads = totalram_pages / (8 * 8192 / 4096);

becomes

max_threads = totalram_pages / 16 ;

Calculating totalram_pages is a bit tricky since its not just the full system ram allocated to pages. The easiest way to calculate it from /proc/zoneinfo, by summing the spanned pages:

cat /proc/zoneinfo | grep spanned | awk ‘{totalpages=totalpages+$2} END {print totalpages}’
3407872

So on my server with 12 gigabytes of memory we’ve got:

max_threads =  3407872 / 16;

becomes

max_threads = 212992;

Now the default maximum number of threads is designed to be able to only consume half the memory from threads alone. But one user should also not be able to consume all of the threads on the system. Back in kernel/fork.c we have:

init_task.signal->rlim[RLIMIT_NPROC].rlim_cur = max_threads/2;
init_task.signal->rlim[RLIMIT_NPROC].rlim_max = max_threads/2;

So there you have it, the ulimit -u for nick (or root) is now set to 106496 procs (threads). nick is unable to take up all of the processes on a box, even using ulimit, he cannot raise past max_threads/2 (106496, though root can). Unless modified via /proc/sys/kernel/threads-max, a maxed out system will not use more than 50% of the memory to store thread structures. Hopefully you too now know a little bit more about linux process limits! Big thanks to colleague and friend Jim Hull for his troubleshooting and major help last night.

The exact formulas may vary kernel to kernel, my findings were through the kernel 2.6.37 source and verified on RHEL5′s heavily modified 2.6.18-194 kernel

If you look at the biggest websites and internet applications, you can pretty much divide them into two groups. Those that are geographically diverse and those that aren’t. It’s kinda shocking that in 2010, the majority of major internet properties still are located in a single (large) datacenter. Though to be fair there is a good reason for that, geo-diversity has many challenges. Problems like directing traffic to the fastest/closest/cheapest/most available location are pretty easy to solve: most people go with BGP AnyCast, targeted DNS responses, or a combination of both. The real challenge though is making sure your actual served content is coherent among all the locations. It would be terrible for a user to upload a photo, sent the URL to their friends, only for the friends to see nothing or worse, the wrong image.

For static content, this is easy, even RSYNCs will be scalable to push out changes to your content amongst your farm. User uploaded content is quite a bit trickier. Within a single datacenter you can efficiently (though not always affordably) solve this using shared storage, iSCSI or NFS. Then applications pretty much can work as if they’re on a single server, session management can be tackled by using cookie or host persistence on the load balancers to make sure a user stays on the same server. What about servers in different locations though? NFS and iSCSI will not be terribly effective over transit.

You will have to push content between your locations then. If you are trying to geographically distribute your own application, you would just write functionality in to immediately push any user uploaded content out to other locations as its created.  Google/Youtube are great examples of this. When you hit content they’ve hosted, it isn’t even hosted on every server, and they direct you to the closest server that has said content. If that content isn’t available locally to you yet, or at all, they can stream it over their own fiber backhaul and out your closest Google POP.

But what if you are hosting a variety of 3rd party software. To my knowledge none of the popular blog packages, forum software, etc has any sort of geo-diversity designed into them. You could of course fork them and write your own, but then you end up supporting N different software packages for your N clients, not affordable or reasonable.  Rsync would do this task but unfortunately it is very intensive and doesn’t scale particularly well because it md5′s your entire tree constantly to see if things changed. As your content scales, the rsyncs would get slower and slower just seeing if changes occurred, eventually leading to massive delays on syncing out user created content.

In the end, its a cool problem, a problem that not too many people have tackled so far. We came up with our own solution, which I unfortunately probably shouldn’t disclose. I wrote the basis of the software last month, though it still needs some bug fixes, testing and more modules to be written for it. It is a difficult problem to tackle, but having worked in telecommunications, no facility is bullet proof, no power is bullet proof, no connectivity is bullet proof, no hardware is bullet proof: geo-diversity is a must going forward in this highly demanding world where everyone expects connectivity and content 24/7