What’s in my files? Heads, tails, cats and more.

As soon as you start working on the Linux command line, you have to start working with files.  Linux follows a very powerful design philosophy expressed as everything is a file. This can take some getting used to, but is incredibly useful once you get it.  Because once you’ve learned how to read and manipulate text files, you can do pretty much anything on your machine.

The first command you need to know is cat. Cat is short for ‘concatenate’, and is used for writing text to and from files. So if I have a file in my current working directory, I can get its contents with cat:


$ echo "hello world" > myfile.txt
$ cat myfile.txt 
hello world

The first line created the file myfile.txt and directed the string “hello world” into it. The second line used cat to read the contents of the file.

Here’s a more real-world example

$ cat /var/log/syslog

This prints out all the lines in the syslog file, present on most unix systems. Don’t worry too much about the contents of the file, it’s probably mostly log records from your system booting up. Now, if you’re inspecting a log file, very often you only want the last few lines, rather than the entire thing. Linux provides a useful command for this very purpose

$ tail /var/log/syslog

Tail, as you might guess, prints out the ‘tail’, or last few lines, of a file. Its companion is head, which prints the first few lines

$ head /var/log/syslog

If you want the ability to view large amounts of text and scroll through it, then the command you want is ‘less‘.

$ less /var/log/syslog

This allows you to scroll up and down through your file. Press q to return to the command line.

Here’s one last neat little trick. I mentioned that on Linux, everything is a file. Even the state of the system is represented as a file.


trevor@vm2:~$ cat /proc/cpuinfo 
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 70
model name	: Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
stepping	: 1
microcode	: 0xf
cpu MHz		: 2294.218
cache size	: 6144 KB
...

Rather than having many different tools for different tasks, a Linux system is more like a box of Lego. The same tools can be plugged into all sorts of different places to enable you to do what you want. I use the humble cat command on a daily basis to modify system settings, read files, and even monitor the health of my machines.

Read the rest of my ‘technology toolkit’ posts:


apt-get: Making your ubuntu machine more better

I pretty much live on the Linux command line.  There’s a number of tools I have in my ‘toolkit’ that I use on a daily basis to automate tasks, manage systems and provide features.

apt-get is one of the most important tools to understand if you’re using an Ubuntu distribution.

apt-get is like the Apple App Store, except it’s been around for much longer and everything it provides is absolutely free. When I’m setting up a new Linux machine I immediately download and install several useful packages.


$ sudo apt-get install ipython
$ sudo apt-get install nmap
$ sudo apt-get install mercurial

and so on.  Very quickly your new Linux machine can be a database server, a graphic design workstation, or a development engine.

Several apt-get commands are worth knowing

photo

$ apt-get install nmap, for example, installs a powerful port-scanner.

$ apt-get remove nmap removes the package we just installed.

$ apt-get upgrade upgrades all installed packages to the latest version

Even more powerfully,

$ do-release-upgrade upgrades your Ubuntu system when a new one is released.   To find out what version of Ubuntu you’re running, I suggest $ lsb_release. On my machine it gives the following output:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 13.10
Release: 13.10
Codename: saucy

Check http://www.ubuntu.com/ to see what the latest release version is.

Have fun!

 

 

More Technology Toolkit Posts


‘top’ and friends – what’s going on on my system?

Linux gives you all sorts of ways of keeping an eye on what’s going on your server.  Here are several that I find useful.  Read my post on apt-get to get any of these that may not be installed on your computer.

The grand-daddy of them all is top

This is like Task Manager on Windows or Activity Monitor on Mac.  It gives you a quick list of all the processes that are running on your computer, how much RAM and processor power they’re using, and how long they’ve been running for.

top, however has a pretty ugly output format, so it’s good that there are several better alternatives.

My preferred tool is htop, which gives a better visual display, including a nice bar-chart showing CPU usage on all your cores, and the ability to sort by CPU, memory usage etc.


  CPU[||                                                                          1.3%]     Tasks: 44, 46 thr; 1 running
  Mem[|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||  332/987MB]     Load average: 0.00 0.01 0.05 
  Swp[|                                                                       0/1021MB]     Uptime: 1 day, 13:00:40

  PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
24133 trevor     20   0 26420  2756  1456 R  0.7  0.3  0:00.57 htop
 1030 root       20   0 91728  4412  3572 S  0.0  0.4  2:21.86 /usr/sbin/vmtoolsd
 1051 postgres   20   0  129M  5056  3628 S  0.0  0.5  0:24.74 postgres: writer process

So what’s making my disk thrash?

htop tells us what processes are using memory and processing power. To see what’s using I/O, use iotop. This needs to be run as root:


$ sudo apt-get install iotop
$ sudo iotop

which gives you a list of which processes are reading and writing to disk.

But what’s using my network connection?

Likewise, we can monitor network usage using iftop


$ sudo apt-get install iftop
$ sudo iftop

which shows the open connections on your machine and how much data is being passed through them.

An alternative to iftop is nethogs which shows you which processes are responsible for data being passed over your network connections.

I also find netstat useful in these situations. netstat can do a lot, but one way I find myself using it a lot is

 $ netstat -plnt 

which means ‘show me all the processes on this machine that are listening for TCP/IP connections, and which port they are listening on’. This is very helpful when you’re setting up a machine and you want to verify that, say, your webserver or database server is running correctly.

Linux gives you many, many ways to see what’s going on on your system, and we’ve only scratched the surface her. Check out the rest of my technology toolkit posts:


What are my servers doing? ping, mtr and nmap

If you’re a network administrator or web developer of any kind, you are often going to want to know what your machines are doing.  Is your database machine connected to the network?  Is your firewall working?  Is your web server properly configured?

ping is the very first tool that we reach for.

  ping simply sends a message to another machine to see if it is responding.

$ ping google.com

When you run this, you should start seeing responses from google.com getting printed out at your command line. If you don’t get this, then your network connection is probably down. This can also be used to see if machines that you own are up and running.

$ ping mywebserver

If ping doesn’t work, and you’re sure that you are connected to the network, the next thing to do is to make sure that you have the right machine name and that your client machine can look up its IP address correctly. That’s where dig comes in.

$ dig mycoolsite.ca 

Check that this gives you the right IP address. Be aware that some DNS providers such as OpenDNS may give you back an IP address even for sites that don’t exist, but it won’t be anything useful.

mtr is like ping on steroids – it will show you all the hops that your network packets are taking between you and your target machine, and how responsive each intermediate machine is being. This is a very handy tool for analysing connectivity problems.

My server is up, but what is it doing?

nmap is a vital tool in every network administrators toolkit. It deserves an article all to itself, but here’s a quick taste:

trevor@vm2:~$ nmap localhost 

Starting Nmap 6.40 ( http://nmap.org ) at 2013-11-28 18:00 PST
Nmap scan report for localhost (127.0.0.1)
Host is up (0.000094s latency).
Not shown: 997 closed ports
PORT     STATE SERVICE
22/tcp   open  ssh
80/tcp   open  http
5432/tcp open  postgresql

Nmap done: 1 IP address (1 host up) scanned in 0.04 seconds

This tells me that machine vm2 is running ssh, a webserver, and the postgresql database engine. If this machine was visible to the outside world, now would be the time when I would make sure that postgresql wasn’t accepting non-local connections.

As with all these tools, nmap can be installed via

$ sudo apt-get install nmap

Find out how to install packages, keep an eye on your Linux system and other useful tips in my technology toolkit posts: