Andrey's Blog

Linux: Network Applications and Services

Table of Contents

  1. How Network Services Work
  2. Talking to Servers by Hand: telnet and curl
  3. Network Servers
  4. Secure Shell (SSH)
  5. SSH File Transfer: scp, sftp, rsync
  6. Diagnostic Tools
  7. Port Scanning: nmap
  8. Network Security
  9. Cheat Sheet

1. How Network Services Work

A network service is a server process that listens on a port and answers clients. The client connects to the server’s IP and port; the kernel hands the connection to the process that owns that port.

        Client host                                 Server host
┌──────────────────────────┐              ┌──────────────────────────────┐
│ curl / browser / ssh     │              │ nginx        listens :80/:443│
│                          │   TCP        │ sshd         listens :22     │
│ 192.168.1.10:53122  ─────┼──────────────┼─► 192.168.1.20:22 ──► sshd   │
│ (random ephemeral port)  │  connection  │                              │
└──────────────────────────┘              └──────────────────────────────┘

Most application protocols sit on top of TCP (HTTP, SSH, SMTP), a few on UDP (DNS, DHCP). Many older ones, like HTTP/1.1 and SMTP, are plain text, so you can talk to them by hand.


2. Talking to Servers by Hand: telnet and curl

telnet

telnet opens a raw TCP connection, so you can type a protocol by hand:

telnet example.org 80
Trying 93.184.215.14...
Connected to example.org.
Escape character is '^]'.
GET / HTTP/1.1              ◄── you type this
Host: example.org           ◄── and this
                            ◄── empty line ends the request
HTTP/1.1 200 OK             ◄── server's response
Content-Type: text/html
...

curl

curl speaks HTTP(S) and many other protocols for you.

curl http://www.example.org/                    # print the response body
curl -I https://example.org                     # headers only (HEAD request)
curl -v https://example.org                     # show request/response headers + TLS handshake
curl --trace-ascii trace_file http://www.example.org/   # full trace of everything sent/received
curl -L https://example.org                     # follow redirects
curl -o page.html https://example.org           # save to a file
curl -X POST -H 'Content-Type: application/json' \
     -d '{"name":"test"}' https://api.example.org/items
curl -k https://localhost:8443                  # ⚠️ skip certificate verification (testing only)
curl --resolve example.org:443:10.0.0.5 https://example.org   # force a host → IP (test ingress/TLS)

--resolve is handy in a homelab: you can test an Ingress or a TLS certificate before DNS points at the server.


3. Network Servers

Common server daemons (the d stands for daemon):

DaemonServiceDefault port(s)
httpd, apache2, nginxWeb servers80, 443/tcp
sshdSecure shell22/tcp
postfix, qmail, sendmailMail servers (SMTP)25, 465, 587/tcp
cupsdPrint server631/tcp
nfsd, mountdNetwork filesystem (file sharing)2049/tcp
smbd, nmbdWindows file sharing (Samba)445, 139/tcp; 137–138/udp
rpcbindRPC portmap service (needed by older NFS)111/tcp+udp

On systemd systems these run as service units:

systemctl status sshd           # is it running? (Debian/Ubuntu: ssh)
sudo ss -tlnp                   # which processes listen on which ports

Socket activation

Instead of running all the time, a service can be started on demand. systemd listens on the port itself (a .socket unit) and starts the .service only when a client connects. This is the modern version of the old inetd/xinetd “super-server”.

  client connects to :22
           │
           ▼
 ┌───────────────────┐   first connection   ┌────────────────────┐
 │ sshd.socket       │ ───────────────────► │ sshd.service       │
 │ (systemd listens) │   starts service,    │ (handles the conn) │
 └───────────────────┘   hands over socket  └────────────────────┘
systemctl list-sockets

4. Secure Shell (SSH)

SSH gives you an encrypted remote shell, command execution, file transfer and tunnels over a single TCP connection (port 22).

Two kinds of keys

┌───────────────────────────────┐            ┌───────────────────────────────┐
│ Client                        │            │ Server                        │
│                               │            │                               │
│ ~/.ssh/id_ed25519      (priv) │            │ /etc/ssh/ssh_host_*_key (priv)│
│ ~/.ssh/id_ed25519.pub  (pub) ─┼── copied ─►│ ~/.ssh/authorized_keys        │
│                               │  once      │   = who may log in            │
│ ~/.ssh/known_hosts ◄──────────┼─ host key ─┤ /etc/ssh/ssh_host_*_key.pub   │
│   = servers I trust           │  on first  │   = proves server identity    │
│                               │  connect   │                               │
└───────────────────────────────┘            └───────────────────────────────┘

The SSH client

ssh user@host                       # log in
ssh -p 2222 user@host               # non-default port
ssh user@host 'df -h'               # run one command and return
ssh -i ~/.ssh/homelab user@host     # use a specific key
ssh -v user@host                    # verbose: debug connection/auth problems

Key-based login

ssh-keygen -t ed25519 -C "andrey@laptop"   # create a key pair (set a passphrase!)
ssh-copy-id user@host                      # append your public key to the server's authorized_keys
ssh-add ~/.ssh/id_ed25519                  # load key into ssh-agent (type passphrase once)

Permissions matter: SSH ignores keys if ~/.ssh is not 700 or authorized_keys/private keys are group/world-writable.

Client config: ~/.ssh/config

Host t490
    HostName 192.168.1.20
    User andrey
    IdentityFile ~/.ssh/homelab
    Port 22

Host *
    ServerAliveInterval 60

Now ssh t490 is enough, and scp, rsync and git use the same alias.

Hardening sshd (/etc/ssh/sshd_config)

PasswordAuthentication no        # keys only
PermitRootLogin no               # or prohibit-password
PubkeyAuthentication yes
AllowUsers andrey

Prefer drop-in files in /etc/ssh/sshd_config.d/. Check the config and reload:

sudo sshd -t                     # test config syntax
sudo systemctl reload sshd

⚠️ Keep your current session open while testing a new config, and verify key login works from a second terminal before disabling passwords.

Tunnels (port forwarding)

Local forward:  ssh -L 8080:localhost:3000 user@server

  your laptop                          server
┌──────────────────┐   encrypted    ┌──────────────────────┐
│ localhost:8080 ──┼═══ SSH :22 ════┼──► localhost:3000     │
│ (browser here)   │                │    (e.g. Grafana,    │
└──────────────────┘                │     only local)      │
                                    └──────────────────────┘

Remote forward: ssh -R 9000:localhost:3000 user@server
  → server:9000 reaches port 3000 on YOUR machine

Dynamic (SOCKS): ssh -D 1080 user@server
  → browser proxy localhost:1080 sends all traffic out via the server

Local forwarding is a quick, safe way to reach a dashboard that only listens on 127.0.0.1 on a remote machine, without opening a firewall port.

ssh -N -L 3000:localhost:3000 t490      # -N = no shell, tunnel only
ssh -J jumphost user@internal-host      # hop through a bastion/jump host

fail2ban

fail2ban watches log files or the journal for repeated failed logins and temporarily bans the source IP with a firewall rule.

 sshd log: "Failed password for … from 203.0.113.9"  (×5 within 10 min)
        │
        ▼
 ┌──────────────┐  matches filter   ┌──────────────────────────────┐
 │  fail2ban    │ ────────────────► │ firewall rule: drop          │
 │  (jail:sshd) │   maxretry hit    │ 203.0.113.9 for bantime      │
 └──────────────┘                   └──────────────────────────────┘

Configure in /etc/fail2ban/jail.local (never edit jail.conf, which updates overwrite):

[sshd]
enabled  = true
maxretry = 5
findtime = 10m
bantime  = 1h
sudo fail2ban-client status
sudo fail2ban-client status sshd             # banned IPs
sudo fail2ban-client set sshd unbanip 1.2.3.4

With key-only SSH logins, fail2ban mostly reduces log noise, but it’s still a useful extra layer.


5. SSH File Transfer: scp, sftp, rsync

scp

scp copies files over SSH. Syntax is like cp, with user@host: in front of a remote path.

scp user@host:file .                         # remote → local (current directory)
scp file user@host:dir                       # local → remote
scp user1@host1:file user2@host2:dir         # remote → remote
scp -r project/ user@host:~/                 # copy a directory recursively
scp -P 2222 file user@host:                  # port (capital P! ssh uses -p)

sftp

Interactive file transfer, like an FTP client but over SSH:

sftp user@host
sftp> ls
sftp> get remote.txt
sftp> put local.txt
sftp> bye

Graphical file managers can often open sftp://user@host/path directly.

rsync

rsync only transfers differences, so it’s ideal for backups and repeated syncs.

rsync -avz src/ user@host:/backup/src/       # archive, verbose, compress
rsync -avz --delete src/ user@host:/backup/  # ⚠️ also delete files removed from src
rsync -avzn src/ user@host:/backup/          # -n = dry run, check first!

⚠️ Trailing slash matters: src/ copies the contents of src, src copies the directory itself.


6. Diagnostic Tools

netstat and ss

netstat reports connections and listening ports. It’s deprecated in favor of ss from iproute2, which accepts nearly the same options.

OptionDescription
-tTCP port information
-uUDP port information
-lListening ports only
-aEvery active port (listening + connected)
-nNo name lookups (faster, and works when DNS is broken)
-pShow the owning process (needs root for other users’ processes)
-4, -6Only IPv4 or IPv6
ss -tlnp              # what listens on TCP, and which process (= netstat -tlnp)
ss -tunap             # all TCP+UDP sockets with processes
ss -nt                # established TCP connections
ss -s                 # summary statistics

lsof for network sockets

lsof treats network connections as open files:

sudo lsof -i                 # all network connections
sudo lsof -i :443            # what uses port 443
sudo lsof -i tcp -sTCP:LISTEN
sudo lsof -i @192.168.1.20   # connections to/from a host
sudo lsof -iTCP -P -n        # numeric ports and hosts (faster)

tcpdump

tcpdump captures and prints packets on an interface, like a command-line Wireshark.

  Network ──► NIC ──► kernel ──┬──► normal delivery to applications
                               │
                               └──► copy to tcpdump (packet filter)
                                        │
                                        ▼
                                   terminal / capture.pcap
sudo tcpdump -i any                          # all interfaces
sudo tcpdump -i wlp0s20f3 -n                 # one interface, no DNS lookups
sudo tcpdump -n port 53                      # DNS traffic
sudo tcpdump -n host 192.168.1.20 and tcp    # combine primitives
sudo tcpdump -n 'tcp port 80 or tcp port 443'
sudo tcpdump -n net 10.42.0.0/16             # k3s pod network traffic
sudo tcpdump -A -n port 80                   # print payload as ASCII
sudo tcpdump -w capture.pcap port 443        # save for Wireshark
tcpdump -r capture.pcap                      # read a saved capture

tcpdump primitives

PrimitivePacket specification
tcpTCP packets
udpUDP packets
ipIPv4 packets
ip6IPv6 packets
port portTCP and/or UDP packets to/from port
host hostPackets to or from host
net networkPackets to or from network

Combine with and, or, not (and parentheses in quotes). Add src or dst to restrict direction: src host 10.0.0.5, dst port 22.

Reading an output line:

16:41:07.512843 IP 192.168.1.10.53122 > 192.168.1.20.22: Flags [S], seq 2810..., length 0
└──────┬──────┘    └───────┬────────┘   └──────┬──────┘        └┬┘
   timestamp         source IP.port      dest IP.port      TCP flags
                                                   S=SYN  .=ACK  P=PUSH  F=FIN  R=RST

A stream of [S] with no replies means packets are being dropped, often by a firewall. An immediate [R] (reset) means nothing is listening on that port.

netcat (nc)

nc reads and writes raw data over TCP or UDP. It works as a client or a server.

nc -zv host 22                     # is the port open? (-z scan, -v verbose)
nc -zv host 20-25                  # small port range
nc host 80                         # talk to a service by hand (like telnet)
nc -l 9999                         # listen on port 9999 (simple server)
nc -u host 53                      # UDP

Quick connectivity test between two machines:

 host A:  nc -l 9999                  host B:  nc hostA 9999
          (waits)              ◄────          type text, it appears on A

Simple file transfer: nc -l 9999 > file on the receiver, nc host 9999 < file on the sender (unencrypted!).

Different netcat versions (OpenBSD nc, ncat from nmap) have slightly different flags; check man nc.


7. Port Scanning: nmap

nmap discovers hosts and the ports/services they expose. Useful for checking your own machines from the outside: what does the network actually see?

nmap 192.168.1.20                  # top 1000 TCP ports
nmap -p 22,80,443 192.168.1.20     # specific ports
nmap -p- 192.168.1.20              # all 65535 TCP ports
nmap -sV 192.168.1.20              # detect service versions
nmap -sn 192.168.1.0/24            # host discovery only (who's online?)
sudo nmap -sU -p 53,123 host       # UDP scan (slow, needs root)
sudo nmap -O host                  # guess the operating system

Port states:

StateMeaning
openSomething is listening and accepted the connection
closedHost reachable, but nothing listens (got a RST)
filteredNo answer: a firewall is dropping packets

8. Network Security

Basic principles

  1. Run as few services as possible. Every listening port is attack surface.
  2. Block as much as possible with a firewall. Default-deny inbound, open only what’s needed.
  3. Track the services you offer to the internet. Know what’s exposed; check with ss -tlnp and an outside nmap.
  4. Use long-term support (LTS) distribution releases for servers, and install security updates.
  5. Don’t give an account to anyone who doesn’t need one.
  6. Avoid installing dubious binary packages from untrusted sources.

Audit loop

     ┌─────────────────────────────┐
     │ 1. What listens?            │   sudo ss -tlnp
     └──────────────┬──────────────┘
                    ▼
     ┌─────────────────────────────┐
     │ 2. Is it needed?            │   no → systemctl disable --now svc
     └──────────────┬──────────────┘
                    ▼
     ┌─────────────────────────────┐
     │ 3. Bound to the right IP?   │   127.0.0.1 if only local use
     └──────────────┬──────────────┘
                    ▼
     ┌─────────────────────────────┐
     │ 4. Firewall allows only it? │   firewall-cmd --list-all
     └──────────────┬──────────────┘
                    ▼
     ┌─────────────────────────────┐
     │ 5. Verify from outside      │   nmap from another host
     └─────────────────────────────┘

More practices


9. Cheat Sheet

TaskCommand
Talk to a TCP service by handtelnet host 80, nc host 80
Test a TLS serviceopenssl s_client -connect host:443
HTTP headerscurl -I url
Debug an HTTP requestcurl -v url, curl --trace-ascii file url
Test host → IP without DNScurl --resolve host:443:IP https://host
Log in / run a commandssh user@host, ssh host 'cmd'
Create key and copy itssh-keygen -t ed25519, ssh-copy-id user@host
Local tunnelssh -N -L 8080:localhost:3000 host
Jump hostssh -J bastion user@internal
Test sshd configsudo sshd -t
Copy filesscp file user@host:dir, scp user@host:file .
Sync directoriesrsync -avz src/ host:dst/ (-n dry run)
Listening ports + processessudo ss -tlnp (old: netstat -tlnp)
Who uses a portsudo lsof -i :443
Capture packetssudo tcpdump -n -i any port 53
Save capture for Wiresharksudo tcpdump -w file.pcap
Is a port reachable?nc -zv host 22
Scan your own hostnmap -sV host
Find hosts on the LANnmap -sn 192.168.1.0/24
fail2ban statussudo fail2ban-client status sshd