Skip to content

The Linux filesystem

Objectives

At the end of this self-learning lab, you should be able to:

  • Understand the concepts of the Linux filesystem, such as paths, cwd, home directory
  • Use some basic file manipulation commands like cp, rm, etc.
  • Understand the permission model in Linux
  • Find and install apt packages

The file tree

  • The file structure of Linux is a big tree of files, where directories contain files and subdirectories.
    • Strictly speaking, directories are also a type of files, and the ones where we can read/write are called "regular files".
  • Unlike Windows, Linux only has one root called /. There are no "drives" in Linux; other filesystems are "mounted" as a directory under the / tree.
  • An absolute file path is the path from / to the file. For example, /home/m2 means that:
    • In / there is a directory called home, which we denote as /home
    • In /home there is a directory called m2, which we denote as /home/m2
  • A relative file path is a path starting from the "current working directory" (cwd) instead of /. It is the same as an absolute path except it does not start with /.
    • If the cwd is /, absolute path is just / + relative path.
  • In a path, a component . means "current directory", and a component .. means "parent directory", i.e. the directory containing it.
    • For example, /home/./m2/catkin_ws/.. is equivalent to /home/m2 because the ./ does nothing and the .. deletes the last component.
    • Hence, we can reference / from any cwd if we ../ many times. For example, ../../../ is eqiuvalent to / if the cwd is /home/m2/catkin_ws.
  • All characters except / (and the NUL byte) are allowed in file paths. However, in general, avoid using any characters other than alphabets, digits, - and _. In particular, avoid using spaces, because some poorly written tools will expand the filename into two separate arguments.

Note

.. does not always mean we can delete the last component in the path. There exists something called "symbolic link" (symlink) in Linux, where a file will redirect to another path. If /home/m2/catkin_ws is a symlink to /home/m2, the example above would actually resolve into /home/m2/.., which is /home.

Read man 7 symlink to know more about symlinks. Symlinks are not often used in M2, but they are fundamental for programmers using Linux.

Many things in Linux can be abstracted as a file!

  • You can access the data on another server if a directory is mounted with sshfs.
  • You can read the signals from an external device by reading from a "device file".
  • You can talk to some processes by writing your input to a "unix socket file".
  • Some files behave differently depending on who is accessing it.
    • The /proc/self/cmdline file always displays the command you are running.

Home directory

A normal user has a "home directory", which is usually /home/<username>. For example, /home/m2 is the home directory of the user m2.

In a Unix shell, the home directory is denoted as ~. For a shell run by the user m2, ~/catkin_ws is converted into /home/m2/catkin_ws.

Caution

This only works on a shell. Do not use this when using subprocess functions in Python, otherwise you will have to deal with a file literally called ~.

On the command line

Check what these commands do!

Hint: All the commands below, except cd, have a manpage with the same name, e.g. man pwd.

  • pwd
  • ls
    • ls -l
    • ls -A
      • Hint: Files whose name start with . are not displayed in ls by default.
    • ls -R
    • ls /
    • ls -l /
  • mkdir
  • rm
    • rm -r
  • mv
    • Hint: mv does not have a -r argument. Moving a whole directory is as fast as moving a single file if you do not cross filesystem boundaries.
  • cp
    • cp -r
  • cat
  • stat
  • cd
    • cd is a shell command. Use help cd instead of man cd.

Command option conventions

Command arguments are always space-delimited. If you want to have spaces in an argument, you have to wrap the argument in "".

For most Linux commands, you can provide optional command arguments in the following syntax:

  • Single-character flags without values: e.g. -l and -A in ls.
    • You can combine them together, e.g. ls -lA.
  • Single-character flags requiring a value: e.g. -m in git commit.
    • You have to put an argument after the option; usually the space between the flag and the value is optional.
    • e.g. Both git commit -mmessage and git commit -m message work.
  • Long flags without values usually start with --, e.g. --copy-contents in cp.
  • Long flags that require a value: e.g. --message in git commit.
    • You must put a space or = between flag and value.
  • Long flags with an optional value: e.g. --backup in cp.
    • If you provide a value, you must use = between flag and value.

However these are just conventions. Some commands such as find do not follow these conventions.

Hint

Sometimes, pressing Tab once/twice can also show you the arguments available and autocomplete them if possible. Sometimes otherwise, it shows the files in the current directory (or in the directory you typed if your cursor is in the middle of an argument).

Try it yourself

Try to complete the following tasks: (You only need to use the commands mentioned above)

  1. Change cwd to your home directory.
  2. Print the cwd.
    • Sometimes the shell prints your cwd in front of your command prompt.
  3. List all files in your cwd, including hidden files.
    • Does the file ~/.bashrc exist?
  4. List all users with a home directory in /home
  5. You may notice there is a .bashrc file in your home directory. Copy it to a path ~/try/it/yourself.
    • The system raises an error called "No such file or directory" if you try to create a file in a nonexistent directory.
  6. Move ~/try/it to ~/try it.
    • Hint: you have to use quotes.
    • Hint: if something goes wrong, use the ls command to check what files you created.
  7. View the contents in ~/try it/yourself.
  8. Delete all new files created in this TIY section.
    • Use ls -A ~ to check that you have really reverted everything.
    • Compare the output with the previous ls -A in 3.

Caution

There is no "undo" function in the filesystem! Once you delete a file, it is lost permanently. (Technically, usually it still exists in the "unused space" area of your harddisk, but it is very very hard to recover it)

Permissions

Most permissions in Linux are controlled by whether you can access a certain file.

How to only allow only the user m2 to access a database?

Make the unix socket file only readable/writable by the m2 user.

The permissions of a file are mainly controlled by three attributes:

  • User owner
    • In Linux, many programs have their own users!
    • Use cat /etc/passwd to list all existing users.
      • You can see some other interesting information from the output. Try guessing what they mean!
      • Use man 5 passwd to view the documentation of the format of /etc/passwd.
    • There is a special user called root. It is a special user which has permission to everything.
  • Group owner
    • Groups are used to give multiple users the same privilege.
    • Use the groups command to see the groups you belong to. In particular, you should see that you have the sudo group if you are in the user created when you installed Ubuntu.
  • File mode
    • Is it rwx (readable/writable/executable)?
    • By whom: ugo (user owner/group owner/others)?

Let's see an example:

$ stat ~/.bashrc
  File: /home/sofe/.bashrc
  Size: 3866        Blocks: 8          IO Block: 4096   regular file
Device: fd01h/64769d    Inode: 8009469     Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/    sofe)   Gid: ( 1000/    sofe)
Access: 2020-09-06 12:53:03.509175956 +0800
Modify: 2020-09-01 11:56:56.293747992 +0800
Change: 2020-09-01 11:56:56.309748117 +0800
 Birth: -

In particular, pay attention to the first Access line.

  • Uid and Gid are the user and group owners of the file.
    • Each user has a group dedicated to itself.
  • The string -rw-r--r-- is the file mode.
    • The first character is the file type.
    • The next three indicate whether the file is rwx by the user owner
    • The next three indicate whether the file is rwx by the group owner
    • The last three indicate whether the file is rwx by other users.

What is executable?

Executable files are those that you can run from the command line directly.

There are two types of executable files in Linux:

  • ELF binaries: These files are like .exe files on Windows.
  • Shell/Shebang scripts: Files starting with #! are called "shebang" scripts. The system will execute them using the command specified on the first line. In particular, this is why you see the line #!/usr/bin/env python3 in many python files and #!/bin/bash in many shell scripts.

Note that file modes mean pretty different things for different file types. For example, the x mode for a directory allows users to cd into it.

Note

To run an executable file in the cwd, you must add the ./ prefix in front of the filename, e.g. ./executable_file. Otherwise, the system will try to search the filename in the $PATH, which does not necessarily include the cwd.

Changing file permissions

When you create a file, by default the file has the mode rw-rw-r--, i.e. it is readable by all users!

Note

You are not the only one living in your system! Some processes are secure only because they are confined by file permissions. Do not assume there are no malicious processes on your system.

To change this, you can use the chmod command with this syntax:

$ chmod o-r file

This will remove the r (read) permission of o (others not in the group) to the file.

Similarly, the following command makes a file executable for the current user: 

$ chmod u+x file

The sudo command

How to login as root?

You don't. Instead, you login as a sudoer (a user in the sudo group) and run commands with sudo.

Using sudo is very simple: you simply type sudo in front of the command you use, and input your own password (not the root password!). You will not be asked to type your password again in a short period.

Try it yourself

The /etc/shadow file contains the encrypted password of your user. Let's see what it becomes!

$ cat /etc/shadow
cat: /etc/shadow: Permission denied

Let's try with sudo instead:

$ sudo cat /etc/shadow
[sudo] password for YOURNAME:
... (omitted)

Installing packages

Ubuntu has an official system package manager called apt. Most common packages (such as git, which is taught in the next section) can be installed from apt directly.

To make apt check for updates and download the latest package list, run:

$ sudo apt update

You most likely run apt update automatically when you installed Ubuntu, but always run it again to be sure (and to ensure you get the latest versions).

Finding packages with apt-file

If you know the name of a command but you forget the name of the apt package, you can search it using apt-file.

First install the apt-file package (it does not come by default) and update its index:

$ sudo apt update
$ sudo apt install apt-file
$ sudo apt-file update

Now suppose we want to find the package that contains the command netstat:

$ apt-file search netstat
... (omitted)
ruby-packetfu: /usr/share/doc/ruby-packetfu/examples/ifconfig.rb
sphinx-doc: /usr/share/doc/sphinx-doc/html/_sources/usage/extensions/ifconfig.rst.txt
sphinx-doc: /usr/share/doc/sphinx-doc/html/usage/extensions/ifconfig.html
thefuck: /usr/share/thefuck/thefuck/rules/ifconfig_device_not_found.py
weevely: /usr/share/weevely/modules/net/ifconfig.py
xorp: /usr/lib/xorp/lib/libxorp_fea_ifconfig.so
zsh-common: /usr/share/zsh/functions/Completion/Unix/_ifconfig

Oops, we are looking at stuff we don't want.

Commands from apt are usually installed into /bin, /sbin, /usr/bin or /usr/sbin. Let's try bin/ifconfig:

$ apt-file search bin/ifconfig
net-tools: /sbin/ifconfig

There we go, we have to sudo apt install net-tools to use ifconfig.

Hint

In some setups of Ubuntu, when you type a nonexistent command, the terminal will tell you the package name:

$ car

Command 'car' not found, but can be installed with:

sudo apt install ucommon-utils

Installing other downloaded files

Sometimes you may find some programs online that only distribute Linux versions by downloading raw files instead of telling you to use apt repository packages.

Installing .deb files

Ubuntu 24.04 package manager GUI is slightly buggy. To install deb files, first get the path to the file. It is usually in ~/Downloads, but if you use Firefox and choose "Open file" instead of "Save file" while downloading, the file ends up saving in /tmp/mozilla_${USER}0 or some similar path.

To install such a file, do not use the Install button from a program called "Ubuntu Software". Instead, open command line and type sudo dpkg -i /path/to/your.deb

Installing .tar.gz/.tgz/.tar.bz2 files

First, extract the files to any directory. Conventionally, to install it for a single user, we extract it under a new directory ~/.local/$PROGRAM_NAME.

If the downloaded binary provides a command, it is usually located in the bin subdirectory. You can either add the ~/.local/$PROGRAM_NAME/bin directory to your $PATH (by editing the .bashrc), or symlink the required binaries with the command

$ ln -s ~/.local/$PROGRAM_NAME/bin/$COMMAND_NAME ~/.local/bin/$COMMAND_NAME

On most setups, ~/.local/bin is already in the $PATH. If it is not and you do not want to edit $PATH, symlink it from /usr/local/bin instead of ~/.local/bin.

Using the GUI

Using GUI is for losers

This works, but if you need to install something on robots, there is no GUI installed (not true as of September 2020) GUI is hard to access.

Using the tar command

As shown in the comic in the last article, the tar command is very tedious to use. Just recite tar xzf $FILE for extracting .tar.gz/.tgz for extracting .tar.gz and tar xjf $FILE for extracting .tar.bz2 files.

Info

The tar command is required by the Linux Filesystem Hierarchy Standard, so it is always available on any compliant Linux distribution.

Using the 7z command

7z is not installed by default. You can install it through

sudo apt install p7zip-full

7z is a much simpler command that works with most archive/compression formats. To extract an archive, use 7z x $FILE, and to create an archive, use 7z a $FILE. It will automatically choose the correct algorithm using your file extension.

Note that 7z x only extracts one level by default, so .tar.gz only gets turned to .tar and still not completely extracted. Run 7z x file.tar again to actually extract the contents.