Blog.Project13.pl » gnu

Terminal Heroes 8: Memory usage scripts

Ktoso — Wed, 24 Feb 2010 10:54:21 +0000

I have a low-memory VPS running online and thus always have to worry about what uses how much memory… Of course top and my favorite htop are very good tools to check this, but sometimes I just want to get a super simple report if I’m overusing memory or not yet. The bellow scripts (most of them was found online or hacked up from multiple scripts to fit my needs) print simple yet very readable memory usage information.

memused

#!/bin/bash

#if [ $1 = '']; then
# ps auxf | awk '{sum=sum+$6}; END {print sum/1024}';
#else
# ps auxf | grep $1 | awk '{sum=sum+$6}; END {print sum/1024}';
#fi
bean=`cat /proc/user_beancounters` guar=`echo "$bean" | grep vmguar | awk '{ print $4;}'` burst=`echo "$bean" | grep privvm | awk '{ print $5;}'` priv=`echo "$bean" | grep privvm | awk '{ print $2;}'`
let total=guar/256 let used=priv/256 let burst=burst/256
echo "VPS memory usage:"
echo "Used: $used MB"
echo "Total: $total MB"
echo "Burstable to: $burst MB"

Sample output:
VPS memory usage: Used: 296 MB Total: 256 MB Burstable to: 512 MB

memfree

#!/bin/bash

#
# Revised 02-Feb-2007: include kernel memory (kmemsize) in 'used' calculation
# and show percentages in output.
#
BEAN=`cat /proc/user_beancounters`
GUAR=`echo "$BEAN" | grep vmguar | awk '{ print $4;}'`
PRIV=`echo "$BEAN" | grep privvm | awk '{ print $2;}'`
KMEM=`echo "$BEAN" | grep kmem | awk '{ print $3;}'`
let TOTL=$GUAR/256
let KMMB=$KMEM/1048576
let PVMB=$PRIV/256
let USED=$KMMB+$PVMB
let FREE=$TOTL-$USED
if [ "$FREE" -gt "0" ]; then
let UPER=$USED*100/$TOTL
let FPER=100-$UPER
else
let UPER="100"
let FPER="0"
fi
echo "VPS Memory:"
echo " Total: $TOTL mb Used: $USED mb (${UPER}%) Free: $FREE mb (${FPER}%)"

Sample output:
VPS Memory: Total: 256 mb Used: 302 mb (100%) Free: -46 mb (0%)

memheld

#!/bin/bash

grep oomguarpages /proc/user_beancounters | awk '{s=$2;t=$3;u=$4; {print "VPS Memory Usage\nCurrent Held: " s/256 "MB\nMax Held: " t/256 "MB\nBarrier: "u/256"MB" }}'

Sample output:
VPS Memory Usage Current Held: 160.832MB Max Held: 294.605MB Barrier: 256MB

memps

#!/bin/bash

echo "ps axo pmem,pcpu,comm | sort"
ps axo pmem,pcpu,comm | sort

Sample output:
ps axo pmem,pcpu,comm | sort 0.0 0.0 script 0.0 0.0 script 0.1 0.0 init #... 2.2 0.0 httpd 4.0 0.0 mysqld 7.3 0.0 httpd 8.0 0.0 httpd %MEM %CPU COMMAND

memprivmpages

#!/bin/bash

beans=`cat /proc/user_beancounters | grep priv`
max=`echo $beans | awk '{ print $4;}'`
use=`echo $beans | awk '{ print $2;}'`
let "per=$use*100/$max"
let "mb=$use/256"
let "mmb=$max/256"
echo "privvmpages usage: $mb MB ($per% of $mmb)"

Sample output:
privvmpages usage: 296 MB (57% of 512)
For those wondering, yes I do have memory usage above “100%” in the output of some of these scripts, and it’s partially true. A VPS does usually have some guaranteed memory, and some “burstable” – so you can use more than just the guaranteed memory for some period of time. On the mashine I ran this I have 256 guaranteed and a “total” of 512MB – including the burstable memory.

If you like any of these, feel free to use them. I use them by creating a scripts directory and adding it to the PATH, so I can simply invoke them from anywhere.

PS: For more detailed memory usage information (as graphs etc), I’m using Munin and really recommend it. The graphs it draws are really nice, and it doesn’t inflict heavy load on the server, and it can monitor remote hosts if needed!

MySQLIntegra: dbfiller & wiki

Ktoso — Tue, 09 Feb 2010 23:06:21 +0000

I passed all my exams at my first attempt and should have now some time to relax, right? Well something like that, I’m actually doing quite a lot these days. We’re on the roll with MySQL Integra.

Our server “Kopiec” is somewhat online (running CentOS off course) – we’re able to start/stop if thanks to a tool that’s called “remote insight”, and I’ve put 3 databases on there. MariaDB, InfiniDB and a clean MySQL instalation. We’re somewhat having problems with TokuDB, but that’s going to be our 4th database of interest. If you want to keep track of our test’s you can take a look at: http://mysqlintegra.project13.pl/.

The first step in getting things tested is filling those db’s with some data, and I mean a LOT of data. So we hacked up a more or less quick and dirty python script called dbfiller (source code avaiable on github) that would simulate a very simple table generation scheme (people working on shifts and checking in when they’re finished or start some activity). If you’re interested, our task is specyfied in the /doc/Opis.pdf file (sorry, only in polish). After a few days it’s up and running and we’re moving on to filling up those databases with some gigabytes of data.

Next we’re going to write some SQL Views/Procedures and then we’ll use Apache JMeter to run some performance benchmarks of some interesting data storage engines.

PS: If you’re wondering about my GWT-Crossword project – yeah, it’s still actual, and I’ll get right on it when done with some of the stuff here.
PPS: Yes, an Linux powered phone AT LAST in my hands! The HTC HERO is comming to me really soon – never again shall I use a ShitOS like WindowsMobile 6.5.

Aiming Higher…

Ktoso — Mon, 21 Dec 2009 21:54:47 +0000

…is exactly what I’m currently trying to do. Thus, I’m out of Mediaframe, and got involved in a fascinating and very cool project. Under the lead of Dr Marek Długosz me and 2 of my friends will be tackling the topic of “MySQL Data Storage Engine Comprehension – with focus on Automatics use cases”. It’s really interesting since MySQL is so popular, yet very little people know about the dozens of 3rd party storage engines – some of them are really interesting, just to name a few:

TokuDB http://tokutek.com/
InfiniDB http://www.infinidb.org
percona http://www.percona.com/docs/wiki/Percona-XtraDB:start
MariaDB http://askmonty.org/wiki/index.php/Maria (Maria, PBXT, FederatedX)
IBMDB2I http://dev.mysql.com/doc/refman/5.1/en/se-db2.html
…ane some more…?

A simple comprehension will be the first thing we’ll do for this project… Later on we’ll do some simulated “real life” tests using JMeter. It’s possible that we’ll write some app based on this database later on – but that’s somehow far off at the moment. That said, I’m really happy to be involved in an interesting and open (it’s all open source & free software after all) topic.

Tomorrow I’m going to the university to setup our very own (awesome ;-)) HP ProLiant DL380 server. It has 6 hard drives, thus it should be very interesting to run HEAVY database LOAD tests – can’t wait. We’ll install CentOS 5.4 and think about it’s future then…

(Almost) Automatically convert files into PDF

Ktoso — Mon, 16 Nov 2009 00:56:55 +0000

I’m learning Numerical Analysis right now and a friend of mine found some really nice PS files by dr Marian Bubek ( http://www.icsr.agh.edu.pl/~mownit/mownit.html ). I was really happy to see some *.ps but some people from my fellow students started complaining that they’d rather have pdf’s and not ghostscript files (who knows why they’d do..?). Anyways, they all had “manual” repetitious methods for converting the files – even uploading to somewebsite to have them converted over there “in the cloud”… I though of a slightly quicker and more efficient way, here it is, hope you’ll like it. Ah, there is One “non standard” requirement for this to work, you have to install cups-pdf :-)

wget http://www.icsr.agh.edu.pl/~mownit/mownit.html -r –level 1 -A.ps –cut-dirs 4

mv www.icsr.agh.edu.pl/* .
rm -rf www.icsr.agh.edu.pl/
for file in `dir -f *` ; do
lpr -P 'cups-PDF' $file;
echo "press enter when cups is done printing…" #yeah, i got lazy here, one could watch lpq for changes etc.
read oczekiwanie;
mv /home/ktoso/Pulpit/cups-pdf/template.pdf ./$file;
done;
# ________________
#< Happy hacking! >
# —————-
# \ ^__^
# \ (oo)\_______
# (__)\ )\/\
# ||—-w |
# || ||

AWK and multiple lines from shell command

Ktoso — Sun, 15 Nov 2009 15:11:16 +0000

Let’s say you want to read the ps output inside an AWK script, it’s not so simple since system returns an exit code and not the string you want – it’s being sent to stdout, here is a simple trick to read multiple lines from a shell command inside awk:
awk 'BEGIN{while ("ps aux" | getline){ print "test " $1 " test"}}' /dev/null | more

I’m an RHCT – yay!

Ktoso — Fri, 02 Oct 2009 19:54:53 +0000

Yes, my GNU/Linux skills have been certified and now I can call myself a Red Hat Certified Technician as I’ve today passed the RH-202 exam with a 100% score – which leads me to the idea that I should have tried to tackle RHCE and not RHCT – I’ll do ir eventually someday I guess anyways.

Happy Hacking!

Blender – 04 – Simple Present

Ktoso — Mon, 21 Sep 2009 00:47:34 +0000

With this simple box i’ve learned a little about the “wave” property in blender. This, a little subsurv and smooth subdivision can actually be quite OK for creating a simple ribbon like the one below:The final render looked like this:

Vim Hacking – 1 – vis

Ktoso — Sun, 20 Sep 2009 18:33:42 +0000

Vim is a really amazing text editor (though I’d like to learn Emacs too soon… :-)) and it can be easily expanded to be even more useful. One super cool extension I found is: vis -- it allows powerful sed-like line modification (you could use sed from inside vim, but that’s not really “handy”). Vis is a vim plugin, and as such, it has to be installed before use, here is how to do this (asuming you have vim 7.0+):
echo 'read some about the plugin at: http://vim.sourceforge.net/scripts/script.php?script_id=1195' vim /home/script/location/vis.vba :source %
With this done you can doo stuff like in this video i did:

Watch it on youtube to see useful annotations! If your still new to vim you might want to start with simple editing as shown here: http://www.youtube.com/watch?v=71YTkxUNwmg&NR=1

Terminal Heroes – 6 – Track X events

Ktoso — Sat, 19 Sep 2009 22:53:33 +0000

Another Command Line Fu oneliner I’ve sumbited today:

#show X window events in chosen window

$ xev -id `xwininfo | grep 'Window id' | awk '{print $4}'`
EnterNotify event, serial 13, synthetic NO, window 0×6400053,
root 0x13c, subw 0x640005b, time 42431610, (367,0), root:(369,63),
mode NotifyNormal, detail NotifyVirtual, same_screen YES,
focus NO, state 16
MotionNotify event, serial 13, synthetic NO, window 0×6400053,
root 0x13c, subw 0x640005b, time 42431610, (367,0), root:(369,63),
state 0×10, is_hint 0, same_screen YES
MotionNotify event, serial 13, synthetic NO, window 0×6400053,
root 0x13c, subw 0x640005b, time 42431618, (366,0), root:(368,63),
state 0×10, is_hint 0, same_screen YES

After executing this, click on a window you want to track X Window events in. You could “read” this command like this: “xev will track events in the window with the following -id, which we get by greping window information obtained by xwininfo” :-)

Ok, but why did I need such weird information? Well, I wanted to make a simple video showing vis, a vim plugin etc. But as it is always in vim, keypresses are essential – so I wanted to “show what i type” bigger :-) I’ll post it really soon btw… :-)

The command was posted on commandlinefu

Terminal Heroes – 5 – List internet active apps

Ktoso — Sat, 19 Sep 2009 14:59:01 +0000

A simple yet usefull command-line-fu I made up:

#list all internet active apps

$ netstat -lantp | grep -i establ | awk -F/ '{print $2}' | sort | uniq
firefox
java
pidgin
thunderbird-b

Lookup this command at CommandLineFu
Thanks to HarimaKenji for fixing a little issue with this command.
—update—
CommandLineFu user submited a super short version for this:lsof -P -i -nComandLineFu ownz!