everyone and welcome to the dark art of container monitoring the talk will be as evil as the title quick intro – about me I’m John Luca I work as a software engineer at C stick and I’m a core developer of the open source of troubleshooting tools is dig as you can see we are very original at this dig when it comes to names let’s introduce the problem of today containers are great right why we like them some of the reason why we like them is because they are isolated self-contained simple and lightweight these are all cool stuff but for these exact same reasons seeing inside containers is not easy at all so when you are in your nice containerized infrastructure and you need to monitor and troubleshoot it essentially you will have you will face big challenges that with with traditional infrastructure you don’t have so the entire talk essentially will revolve around the question can monitor and troubleshooting play nice with these properties what we are going to talk about essentially the rest of the talk is going to be a live review of the monitor and troubleshooting options that you have and we’re going to focus on the most popular common line tools then I’m going to quickly go through some advanced namespaces tricks for gigs and then we are going to focus on sis dig which is the tool where you have the most expertise let’s get started for this talk it is okay we are going to use as a Dem infrastructure a web application that is a distributed WordPress application made by seven containers where I have four containers running the WordPress application and they are running Apache serving the PHP code itself the persistent layer is achieved via my sequel container there is then an entry point made by HC proxy that lives in another container and balances the try between before WordPress containers and finally there is a client container that essentially generates some HTTP traffic to keep the application alive to prove you guys that I’m not cheating the infrastructure is up and running I can ping it from my browser and let’s get started what are some things that we’re interested in monitoring I’m going to focus on these five categories which will cover 99% of what one is interested in and we’re going to focus on the resource usage which is an overview of how your system your containers and your processes are doing then we’re going to dig into network and file activity then we’ll switch to system containers and processes errors and then we’ll take a look at some application level monitoring troubleshooting let’s pick the first item Razors huge if I had to choose what is the first tool that I could use for doing general system troubleshooting at a higher level my first choice will be top right so I can run top and of course this resolution is low but I can see all the all the processes in my system essentially a bunch of Apache processes with all the system resources they’re consuming CPU memory as well as the global reserves usage for the machine but this doesn’t play very nice with containers because look here I can look at the common light but essentially have a wall of Apache processes and I cannot easily see on which container these processes live so if I have one Apache process that is consuming 100% of CPU how do I know on which container I’m going to do docker exact next and and troubleshoot I can add some fields to top for example one interesting one is the C groups essentially the C groups are the Linux kernel facility that docker uses to group together all the processes in one container for resource control and if I do this they can see that a new column is added that contains essentially the C group name for each Apache processes now the reserved is a bit low but if you can see your essential you can see that as you go through with the Apache processes the C group changes and this C group is nothing more that the container ID of the docker container itself so you can use this for filtering purposes and to decide what what’s next if one Apache process start misbehaving another thing that I cannot easily say with top is of course how my containers are doing I cannot see the CPU and memory of of each of my each of my containers because I’m

just seeing the processes and another thing that is difficult to spot is some information that live inside containers for example here I just see the pit of the Apache processes as seen outside the containers because I’m running top in my Linux VM but I’m not saying for example the internal pit as seen inside of the container which is an information that you need to use if you need to do some intrical Taner process communication so for doing these things I can use some docker specific tool as you’ve seen with docker PS I can essentially see the list of my containers running along with their statuses but what we can do better than that for example with docker stats I can pass a bunch of container names and now I get a real-time update of the system utilization per container and this is very useful because not only it gives me a very good overview of how my containers are behaving but also this is based on a nice REST API that one can use in their own scripts and monitoring tools to get a greater level of visibility and do some custom actions once you see that one container is misbehaving essentially like say that is that is this one then I can zoom in into the container using docker top and I can do docker top WordPress one and this will show me all the processes and this will show me all the processes that live inside this container in this case all the Apache processes and the nice thing of this is that I can use for example a ts like syntax so if I pass a UX now I will see not only the common line but also the CPU usage in the memory usage of all the process in the container so overall this is a pretty nice workflow to get information about the overview of your containers and process in the system the next item in the list where the network connections I’m a big fan of monitoring network connections I do it as often as you can and the program that comes at the beginning to my mind when monitoring connection is netstat right I can do net stat and this will show me all the TCP connections that are happening in my system but immediately we see that here aged I can just see one connection which is one connection on port 22 so it’s the ssh connection between my terminal and my linux vm all the connections that are happening with the docker container so the connections between WordPress and my sequel and they shape rocks in WordPress they don’t show up here so this is pretty bad why is it this is because each container of course gets its own network namespace and its own network stack so each container from a network point of view is completely isolated from the rest of the system each container gets its own network interfaces its own network connections and if I run netstat in the host essentially I can just see the connections that live outside the containers in my oath and as I said this is not good at all the first thing that come to my mind to solve this problem is well I could run netstat inside the docker container right so say that I’m interested in the connections for the my secure container I can go a docker exec and execute essentially a bash shell inside the my secure container and run netstat but oops I get comma not found why because containers according to the previous slide are simple and lightweight the the containers are not loaded with all your tool belt of monitoring tools that you might need and installing all of them is usually consider an anti-pattern in this case I’m lucky because I have a pity get so if I remember the name of the package that contains netstat which I don’t I could easily install it but most of the time this is not possible for example very recently I’ve been seeing this movement where people really try to deploy very little containers where a direct stream is just a single statically link to go application and that’s the entire file system of the container if you’re lucky there’s a busybox runtime to bootstrap but otherwise sometimes you just see a single application and and it actually works and in those cases good luck installing net start with a package manager or something there’s really nothing at all you can do so one possibility might be making a sort of Frankenstein monster where essentially say I have netstat the executable in my host and what if I could run a net stat

against the network namespace of a container that I’m interested in monitoring this is actually possible for example using the system utility that is called NS enter this is a system tool that allows you to run a program inside the namespace of another process this is advanced but it’s also pretty cool so say that I want to see the connections of my my sequel container so first I’m going to grab the pit of my sequel which is three seven nine six and then I go to do NS enter specifying as a target peed three seven nine six and say enter the network namespace and then I’m doing net stat slash and T and now I’m seeing all the connections so this is pretty cool I’m able to see of course that my sequel is having a bunch of connection port 3306 from all the wordpress containers still not ideal because I had essentially to do this for each container that I want to see the connections I still don’t have a tool that shows me an overall list of all the connections in my system but overall you agree with me that this NS enter workflow is horrible right you don’t want to that in production you go crazy so as a next step I want to show you what we did to essentially tackle this problem with this big sis dig is this a fairly young open sewer system troubleshooting tool and you can think of it essentially as a mix of s tres TCP dump l sof with some lua scripting in simplest form sis dig allows you to capture a bunch of system events and those define in a second what I mean by system events and then it allows you to filter them in a very smart way and it allows you to run very useful scripts on them that are written in lua at the we call chisels and most important the most relevant thing to this talk is that is a is a troubleshooting tool that has native support for containers it was designed from day one with support for containers how does it work essentially this is the typical architecture of your host running containers right so you have a single instance of the kernel and then you have a bunch of applications running outside the container and then you have all the containers with their applications running inside 60 quarts by shifting a little current module that intercepts all the activity between applications and kernel these interactions are 99.9 percent of the time system calls so very very very rich information close to your code and by capturing all this information this is then analyzed by the Cystic application inside user space and the nice thing about the Cystic application is that it’s designed to work outside the container as a normal system tool like we’ve seen top Ness that but you can also run in a separate privileged docker container that can inspect the activity of all the other containers in the system by passively monitoring their system call activity so this plays very nice with the principle that we have seen before isolation self-contained and all so this doesn’t fight your workflow even if you run your containers on bare bone distributions such as core OS or whatever you are using for docker machine that they didn’t even have a package manager right and let’s see how this works essentially in its simplest invocation as I said since dig is just we just print me on screen all the system events that are happening in my system so this is hardly useful but as I said one of the first thing that they can do is using filters with dash l I can essentially see all the filters this is big support and they can essentially filter on dozen of filters about file types about process attributes or about event types for example let’s run sis dig with this filter process that name equal to my sequel D and now we see that the output starts getting a little bit more interesting because now we see all these system events just related to sis dig sorry just just related to my sequel in fact I see the process name and actually the container ID and the container name where these events are generated so this is what I meant by native support for containers and if you don’t like this flat list of output that at times it can be very confusing you can choose to run what I what I call before cheeses that are just Lua scripts and we have a huge collections of them and this will essentially take the system events will process them and we’ll just emit a nicer output which in this case for example is the file

activity done on each container broken down by each file so this is for example this is a very powerful metric that falls into the bucket of file activity on my previous list so with this in mind I want to pull it quickly go over how you will address those five points with the system troubleshooting tool like this big designed for containers let’s start with the first one the first one was resource overview I’m going to use CC stick which is just a nice curses UI forces dick and when I run it I see an output it is very similar essentially to top or even better H top if you’re familiar with it but what is important to see is there are two differences one now I’m not only see the processes but I’m seeing also the containers where these processes are leaving so here I have the immediate information of what container I need to look for if I’m looking for trouble and I can filter on that for example I can do WordPress for and now I just see the de processes on the WordPress for container and another thing that is very interesting is that we don’t know we don’t show only the pit of the processes but also the internal page which we call the virtual pit so this is the pit that you will see inside the containers if for example you are logging it somewhere or if you are doing docker exact and you need to send a signal to it so this we thought it was a pretty useful information let’s iterate to the to what was the next item in the list with what’s the connections how would you see the connection Cystic by pressing f2 essentially I can switch view and they can change from the processes view to a bunch of other predefined and customizable views that address pretty much all all sort of areas that you might might be interested in troubleshooting for example with connections now I see the connections pretty much like in the nest at output but notice here if I scroll horizontally now I see also on which container they are made and for which process they are made so finally I ever now put that passively by sitting in a separate container shows me all the connections that are happening inside my hosts regardless of the container and this is a very useful pattern because then you have all these columns for example bite scene or bytes out and you can and you can sort them so that you can just see what you’re interested in and of course you can filter for example you can say filter by HG proxy so you can just see the connection that they CH a proxy in doing is doing which in this case is a bunch of connections to port 80 of all the other WordPress and so this is essentially how you will go on troubleshooting network connections with the container centric tool then as I said I’m going to repeat myself but containers are first-class citizens in this tool so if I want to for example open another view and look at containers now I get an output that is pretty similar to the docker stats one where I see each container along with this metadata and I see an overview of all the activity that is doing in terms of network in terms of CPU in terms of file activity and the good thing is that I can for example click on one say H a proxy and I’m brought it into this view where I just see the processes for this container which in this case there are two one is this Python script which is the entry point because it has DP’d one and then there is the H a proxy application itself that does of course all the business logic more I can see here that H a proxy is doing a bunch of network activity and as I’ve seen before these are all the connections coming from my client and go into WordPress and all that but would it be cool if I could see what’s happening inside those network connections something like people will do a Wireshark or TCP dump right and for this we have this Eco mode that you can enable the pressing f5 it comes very handy and by doing this we can essentially see all the a your buffers exchanged by the h3 proxy application let’s format it in a nicer way and now I can see a more user-friendly the view that I can pause and let’s look for example for something interesting like get underscore and here and now this is for example an isolation of an entire HTTP request where I have one host dot 0.7 that is asking to H a proxy this

HTTP request get slash and then I have a proxy that is reading the request and it’s forwarding it in fact we can see from the HTTP header x-forwarded-for dot 0.7 it’s forwarding it on behalf of my client is forwarding it to the wordpress containers and then you will get the HTTP response and then it will forward it towards again my client so considering that it is all pretty passive and it’s from a separate container it’s pretty cool and can become very handy when you need to do some real production troubleshooting the next item in the list where the files activity so not surprisingly for this we have a files view that will show me for each container and for each file which are the files that are being opened written or read the most all these again by passively monitoring in a separate container and for example let’s sort by bytes out to see what are the files that are being written the most and I can see that seems that pretty much all the WordPress containers are doing a fair constant amount of rise to access that log the Apache log file right and once again I can use the f5 eco mode to see what is actually being written in real time inside the access dot log containers from the host or from a separate container so I like to think as this as a tail command for UNIX but it works everywhere and without doing docker exec or anything from from a central location I can inspect whatever it’s happening and of course these files these files don’t exist in the host because if I just quit now Cystic and I go look for this file doesn’t exist this is in the file this is in the mountain in space of the container so it’s in the separate file system of the container the next item in the list were the errors sometimes it’s good to see system and processes errors so also for this we have a view again container centric which is containers arrow that will show me all the errors categorized by file and network and memory errors for all my containers so I can sort by file and I can see that the WordPress containers have a bunch of file errors so I can zoom in with enter and now I brought in to a view that is showing me the error code that this container is generating which is pretty cool and the most frequent errors seem to be this Ino ant which if you have done some low-level C programming it just means no such file or directory let’s dig more into this error we have this nice mode called f6 dig and by digging into it and brought to the Cystic events that are generating this error so in this case I can see that is the WordPress tree obviously the Apache process that is try to open these files the dot htaccess file pretty much everywhere in the in the directory that is supposed to contain the static assets so in this case this is expected because Apache is supposed to be doing that but again using this workflow you might catch some pretty nasty things and not always being this lucky the last item in the list was the application level activity we have already seen a little bit of that with the f5 eco mode but you can get more interesting and for this I want to introduce one more feature if I want I can not only see the live system events as they are happening in the system but I can choose for example with this common line to save them on a trace file that is completely self-contained and can be analyzed offline at a later time and these are workflow that is borrowed from the TCP dump Wireshark world and it’s proven it’s very effective and once I have this I can read it back and or even better I can use this Cystic and they get the same experience that was getting in real time but on the trace file I can switch the container view I can select one container I can select f5 and see all the activity and this don’t store the trace file this is a very powerful workflow because it means that you can capture anomalous – ation in production and then share it with your coworker or analyze it offline keep it for your records it can become very handy and I want to use this this trace file to show you the last example of application level activity one of the things that I like to see sis digas is a grep for your entire system for example look at this common line where I’m reading my trace file using as a filter

key the buffer contains 404 not found so here I’m saying – sis dig okay show me all the system events that did some IO content that contained 404 not found and by doing this well now the resolution is a bit low but I can see that very often the WordPress for container in the Apache process generated a bunch of HTTP 404 not found responses so what if I want to see more of these to do application level troubleshooting I can choose to not see the flat list of event but use the echo file descriptor chisel for example they will show me the i/o buffers exchange in a user-friendly way and here in fact I can see the full HTTP response done on these connections what if I can look more what happened at the application level and see for example the matching HTTP response I can just take this TCP connection and filter on it so the port was four five three three five and so I can run again the echo file descriptor chisel using a sport five twenty five and now I get the full connection with the HTTP request so we can see that at some point as a proxy forwarded on behalf of my client the non-existent request which not surprisingly failed which was then forwarded to one of the WordPress containers and we got as a response HTTP 404 not found so again this is just example of how you can do from a completely separate container application level troubleshooting this is pretty much all the things that I wanted to show there is one more thing we we have been playing a lot that is big with the system events and we found out that essentially it’s cool if you look them at the host level but it’s even cooler if you correlate them on a cluster level so what happens if you if you fetch all the system events from multiple hosts and then you mix them together in a central location for doing some cool visualization and for these I want to show you part of our software the service monitoring troubleshooting tool which is the cloud with this tool essentially here just by looking at the seasons at the system events we are tracing a complete topological map of a fairly complicated but not not that much infrastructure and here we are seeing essentially clusters where clusters are group of hosts that semantically behave the same and I can see how they are correlated and I can zoom in in one cluster to see the nodes that are that are part of this cluster along with their connection and if I’m not happy I can zoom into one host for example this one that seems red I can zoom into one host and see a map of the containers of the of just this host and these if i zoom in a little bit more you will see that it’s essentially the same infrastructure that I used on my laptop with the same container names because I use the same bootstrap script and essentially RIA you have a mini-map inside the host and if I’m not happy I can zoom in inside each container and see the processes that are leaving its container so there was a cheap proxy with the Python in script DHA proxy of course binary and then all the Apache processes and all these all these is created from the boring list of system events that you’ve seen at the beginning because we see all the connections that are at in the system Co level between processes and we are connecting we are connecting essentially the processes and we’re also connecting processing processes living on different machines and as you can imagine this is not completely easy to do because there’s a bunch of IP tables crap going on when you expose ports and but we managed to do it and so this is just an example that if you have a lot of meaningful data at your fingertips you can do some pretty cool if you want to play with it and this is pretty much the end of what I wanted to show I think I’m out of time but if there is probably a couple questions we can take them the question is if sysd can stream in different outputs other than the system events well there are a couple ways one we support JSON output so you can essentially format the system events in JSON that you can parse as you want or it’s also if you look at the open source project on github you’ll see that it’s made by a bunch of libraries so you can

just extract the one of the library that is used for event inspection and you can hook it up in your framework as you please and then you get direct objects that you can play with your with your favorite programming language so yeah essentially the question is if we can see share filesystem with containers well it depends on how your do the sharing if you do just the simple docker bind mounting or whatever that’s completely supported completely a hundred percent supported and you will see everything you will be able to group by directories and see it exactly as if it was part of the container if you do share file system at the NFS level or something they can become a little bit tricky because some of the interaction of NFS is not handled completely just by system calls but most of the stuff happens in kernel same thing with safes or samba but you can have also pretty good level of visibility there but we just did get the most simple bind mounting you’re under percent covered so is this cluster available just versus the cloud at the moment yes this is part essentially we ship a little a little plugin forces the eaglet allows you to create a compact output that you can stream over the network to sis the cloud and then this visualization is done there so at this moment this map is just part of our commercial product but again the the data is there so if you feel like doing the grunt work and creating it yourself and come up with nice visualization you can do it and please share it ok so talk about performance over at the Cystic essentially the kernel module itself has been designed to be very very very gentle on system system resources it’s used at this point by thousands of servers in production across the world and no one has ever complained so if you use it with the with the kernel module installed there’s no variety of your application but the Cystic application itself is a user space application that as you can see it does a bunch of stuff so the CPU usage of Cystic itself can can also become in the order of percentages of CP of the system but again then it’s up to you creating a client that is more lightweight or something but the kernel module itself doesn’t affect the throughput of your application it’s it’s very well designed so essentially sitting at this moment doesn’t have protocol level visibility so you can just see in plain text what is going on over your network over your network buffers or over your v buffers so it wouldn’t be too easy to see just isolate HTTP requests or something but that being said it’s not too difficult either to write a chisel that will work with cc’s dig and with cystic on which you can write your little HTTP parser very easily and then you will be able to essentially group by all the metrics or something if you go on our on our website there are examples of how to create your own script and it’s very powerful because it will work in the common line but also in CC sticks so you can have this nice table where you can put URL HTTP method I I I take it back I don’t think it would be it would be difficult at all yeah yeah you mean this one this one is completely real-time now I’m seeing it as nap in the past but I can click on go live and I’m seeing see how the numbers are changing and now more nodes are appearing this updated every second with the delay of a couple seconds so we we don’t string to the network all this is big system events because there would be potential in the order of hundreds of megabytes per second we have as I said we have a little plug-in on top of sis dig that summarizes this information in a very compact way and just ships over the network what it takes to compose this visualization which is essentially you can think of it as a table with all the network connections processes and stuff like that and more of course magic stuff that allows you to for example circumvent the IP tables and all the crap yeah yeah yeah yeah yeah yeah you can’t nothing for example I can choose to see the past couple hours and I will see how the infrastructure works for the cup past couple hours again zoom back and I can see how it was from 350 to 550 today it’s yeah we have we have yeah we go to pretty color with the widow in the city card all right I think I’m well over well past my time thank you everyone you