Prepare for DoS like Cloudflare do

I watched nice presentation about how Cloudflare protects itself against DoS. Most of us are not able to do that exactly like them but some of tips were general enough to be used on typical web front server.

I took notes from this presentation and presented here. Thanks to Marek agreement I also reposted all examples (in easier to copy paste way).

Howto prepare against ACK/FIN/RST/X-mas flood

Use conntrack rule:

iptables -A INPUT --dst 1.2.3.4 -m conntrack --ctstate INVALID -j DROP

which will only work with disabled tcp_loose setting (it’s by default enabled) with addition to sysctl:

sysctl -w net.netfilter.nf_conntrack_tcp_loose=0

Howto prepare against SYN floods

SYN flood is hard case – because when you use conntrack it will make your performance worst validating state for every new single packet.

The only way to get around this is to enable syncookies:

sysctl -w net.ipv4.tcp_syncookies=1
sysctl -w net.ipv4.tcp_timestamps=1

Enabling syncookies will cause loose of some of connection informations, that are pretty useful like:

  • window scaling factor
  • ECN bit (Explicit Congestion Notification)

For that we will use tcp_timestamp option, that will use few bits from timestamp field to store some of this informations.

This still may be not efficient enough, but in kernel 4.4 there will be some update to how syncookies are served that should make it few times faster than with older one.

Related docs: https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt

Howto prepare against botnet attack

Symptoms:

  • concurrent connection count going up
  • many sockets in orphaned state
  • many sockets in time wait state

Solutions:

  1. Enable connlimit feature on conntrack to limit amount of concurrent connections to our service
  2. Use hashlimits to rate limit SYN packets per IP
  3. Use ipsets to efficiently block many IP/subnet addresses
    • manual blacklisting – feed IP blacklist from HTTP server logs
    • supports subnets, timeouts
    • automatic blacklisting hashlimits
  4. Disable HTTP keep-alives to make this attack look more like SYN flood

This may still not work against DDoS because huge amount of bots won’t allow you to block them efficiently enough.

Some exciting system tweaks and examples from this presentation

I hope to find some time to merge them into template/script that could be used much easier – but first I have to play with these rules a little and test what will be most useful.

NIC: Discard with flow steering

ethtool -N eth3 flow-type udp4 dst-ip 129.168.254.30 dst-port 53 action -1

Flow steering for priority

ethtool -X eth3 weight 0 1 1 1 1 1 1 1 1 1 1
ethtool -N eth3 flow-type tcp4 dst-port 22 action 0

SYN backlog size

sysctl -w net.core.somaxconn=65535
sysctl -w net.ipv4.tcp_max_syn_backlog=65535

It’s rounded to next power of two (in this case to 65536).

SYN backlog decay

sysctl -w net.ipv4.tcp_synack_retries=1

L7 connection count

sysctl -w net.ipv4.tcp_max_orphans=262144
sysctl -w net.ipv4.tcp_orphan_retries=1

sysctl -w net.ipv4.tcp_max_tw_buckets=360000
sysctl -w net.ipv4.tcp_tw_reuse=1
sysctl -w net.ipv4.tcp_fin_timeout=5

L3: u32

iptables -A INPUT \
  --dst 1.2.3.4 \
  --p udp -m udp --dport 53 \
  -m u32 --u32 "6&0xFF=0x6 && 4&0x1FFF=0 && 0>>22&0x3C@4=0x29" \
  -j DROP

L4: Conntrack

iptables -t raw -A PREROUTING \
  -i eth2 \
  --dst 1.2.3.4 \
  -j ACCEPT

iptables -t raw -A PREROUTING \
  -i eth2 \
  -j NOTRACK

iptables -A INPUT \
  --dst 1.2.3.4 \
  -m conntrack --ctstate INVALID \
  -j DROP

Tuning conntrack

sysctl -w net.netfilter.nf_conntrack_tcp_loose=0
sysctl -w net.netfilter.nf_conntrack_helper=0
sysctl -w net.nf_conntrack_max=2000000
echo 2500000 > /sys/module/nf_conntrack/parameters/hashsize

More info about conntrack sysctl options: https://www.kernel.org/doc/Documentation/networking/nf_conntrack-sysctl.txt

L7: Connlimit

iptables -t raw -A PREROUTING \
  -i eth2 \
  --dst 1.2.4.5 \
  -j ACCEPT

iptables -A INPUT \
  --dst 1.2.3.4 \
  -p tcp -m tcp --dport 80 \
  -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN \
  -m connlimit \
  --connlimit-above 10 \
  --connlimit-mask 32 \
  --connlimit-saddr \
  -j DROP

L7: ipset for blacklisting

ipset -exist create ta_d335c5 hash:net family inet

ipset add ta_d335c5 192.168.0.0/16
ipset add ta_d335c5 10.0.0/8

iptables -A INPUT \
  -m set --match-set ta_d335c5 src \
  -j DROP

L7: being evil – TARPIT

iptables -A INPUT \
  -m set --match-set ta_d335c5 src \
  -j TARPIT

TARPIT target will imitate successful connection for the client (bot in this case) but without responding to it’s queries. It will cost that bot a lot more resources and time to timeout and drop this connection than when using DROP or REJECT here.

l7: hashlimit for rate limiting

iptables -A  INPUT \
  --dst 1.2.3.4 -p tcp -m tcp --dport 80 \
  --tcp-flags FIN,SYN,RST,PSH,ACK,URG SYN \
  -m hashlimit \
  --hashlimi-above 123/sec \
  --hashlimit-burst 5 \
  --hashlimit-mode srcip \
  --hashlimit-srcmask 24 \
  --hashlimit-name 341654b1d4af9bf \
  -j DROP

L7: auto-blacklist

ipset -exist create blacklist hash:net timeout 60

iptables -A INPUT \
  --dst 1.2.3.4 \
  -m set --match-set blaclist src \
  -j DROP

iptables -A INPUT \
  --dst 1.2.3.4 -p tcp -m tcp --dport 80 \
  --tcp-flags FIN,SYN,RST,PSH,ACK,URG SYN \
  -m hashlimit \
  --hashlimit-above 100/sec \
  --hashlimit-mode srcip \
  --hashlimit-srcmask 24 \
  --hashlimit-name hl_blacklist \
  -j SET --add-set blacklist src

L7+: payload in TCP – string

iptables -A INPUT \
  --dst 1.2.3.4 \
  -p tcp --dport 80 \
  -m string \
  --hex-string 486f737777777777... \
  --from 231 --to 300 \
  -j DROP

For more informations and explanations watch this great presentation:

And here is the whole presentation with additional examples:
https://speakerdeck.com/majek04/lessons-from-defending-the-indefensible

Let’s Encrypt – without auto configuration

From the first moment I heard about Let's Encrypt I liked it and wanted to use it as fast as possible. But the more I read how they want to implement it, the more I dislike it.
Current project with automatic configuration is not what I want to use at all. I have many very complicated configs and I do not trust such tools enough to use them. I like UNIX's single purpose principle, tools should do one thing and do it well – nothing more.

But there is one neet tool that use Let's Encrypt API only leaving all configuration for me, it's acme-tiny python based script. I won’t copy/paste examples – documentation is written pretty well.

Generate ECDSA key with OpenSSL

After the last NSA scandal I’ve found some time to read some texts about PFS and ECDSA keys lately. I always used RSA keys but wanted to give a try to ECDSA so I wanted to give it a try (test performance, etc). Here is how I’ve done it.

Firstly find your favorite curve. A short tip about bit length and complexity could be found here. From it you will now that using 256 bit ECDSA key should be enough for next 10-20 years.

$ openssl ecparam -list_curves
  secp112r1 : SECG/WTLS curve over a 112 bit prime field
  secp112r2 : SECG curve over a 112 bit prime field
  secp128r1 : SECG curve over a 128 bit prime field
  secp128r2 : SECG curve over a 128 bit prime field
  secp160k1 : SECG curve over a 160 bit prime field
  secp160r1 : SECG curve over a 160 bit prime field
  secp160r2 : SECG/WTLS curve over a 160 bit prime field
  secp192k1 : SECG curve over a 192 bit prime field
  secp224k1 : SECG curve over a 224 bit prime field
  secp224r1 : NIST/SECG curve over a 224 bit prime field
  secp256k1 : SECG curve over a 256 bit prime field
  secp384r1 : NIST/SECG curve over a 384 bit prime field
  secp521r1 : NIST/SECG curve over a 521 bit prime field
  prime192v1: NIST/X9.62/SECG curve over a 192 bit prime field
  prime192v2: X9.62 curve over a 192 bit prime field
  prime192v3: X9.62 curve over a 192 bit prime field
  prime239v1: X9.62 curve over a 239 bit prime field
  prime239v2: X9.62 curve over a 239 bit prime field
  prime239v3: X9.62 curve over a 239 bit prime field
  prime256v1: X9.62/SECG curve over a 256 bit prime field
  sect113r1 : SECG curve over a 113 bit binary field
  sect113r2 : SECG curve over a 113 bit binary field
  sect131r1 : SECG/WTLS curve over a 131 bit binary field
  sect131r2 : SECG curve over a 131 bit binary field
  sect163k1 : NIST/SECG/WTLS curve over a 163 bit binary field
  sect163r1 : SECG curve over a 163 bit binary field
  sect163r2 : NIST/SECG curve over a 163 bit binary field
  sect193r1 : SECG curve over a 193 bit binary field
  sect193r2 : SECG curve over a 193 bit binary field
  sect233k1 : NIST/SECG/WTLS curve over a 233 bit binary field
  sect233r1 : NIST/SECG/WTLS curve over a 233 bit binary field
  sect239k1 : SECG curve over a 239 bit binary field
  sect283k1 : NIST/SECG curve over a 283 bit binary field
  sect283r1 : NIST/SECG curve over a 283 bit binary field
  sect409k1 : NIST/SECG curve over a 409 bit binary field
  sect409r1 : NIST/SECG curve over a 409 bit binary field
  sect571k1 : NIST/SECG curve over a 571 bit binary field
  sect571r1 : NIST/SECG curve over a 571 bit binary field
  c2pnb163v1: X9.62 curve over a 163 bit binary field
  c2pnb163v2: X9.62 curve over a 163 bit binary field
  c2pnb163v3: X9.62 curve over a 163 bit binary field
  c2pnb176v1: X9.62 curve over a 176 bit binary field
  c2tnb191v1: X9.62 curve over a 191 bit binary field
  c2tnb191v2: X9.62 curve over a 191 bit binary field
  c2tnb191v3: X9.62 curve over a 191 bit binary field
  c2pnb208w1: X9.62 curve over a 208 bit binary field
  c2tnb239v1: X9.62 curve over a 239 bit binary field
  c2tnb239v2: X9.62 curve over a 239 bit binary field
  c2tnb239v3: X9.62 curve over a 239 bit binary field
  c2pnb272w1: X9.62 curve over a 272 bit binary field
  c2pnb304w1: X9.62 curve over a 304 bit binary field
  c2tnb359v1: X9.62 curve over a 359 bit binary field
  c2pnb368w1: X9.62 curve over a 368 bit binary field
  c2tnb431r1: X9.62 curve over a 431 bit binary field
  wap-wsg-idm-ecid-wtls1: WTLS curve over a 113 bit binary field
  wap-wsg-idm-ecid-wtls3: NIST/SECG/WTLS curve over a 163 bit binary field
  wap-wsg-idm-ecid-wtls4: SECG curve over a 113 bit binary field
  wap-wsg-idm-ecid-wtls5: X9.62 curve over a 163 bit binary field
  wap-wsg-idm-ecid-wtls6: SECG/WTLS curve over a 112 bit prime field
  wap-wsg-idm-ecid-wtls7: SECG/WTLS curve over a 160 bit prime field
  wap-wsg-idm-ecid-wtls8: WTLS curve over a 112 bit prime field
  wap-wsg-idm-ecid-wtls9: WTLS curve over a 160 bit prime field
  wap-wsg-idm-ecid-wtls10: NIST/SECG/WTLS curve over a 233 bit binary field
  wap-wsg-idm-ecid-wtls11: NIST/SECG/WTLS curve over a 233 bit binary field
  wap-wsg-idm-ecid-wtls12: WTLS curvs over a 224 bit prime field
  Oakley-EC2N-3: 
	IPSec/IKE/Oakley curve #3 over a 155 bit binary field.
	Not suitable for ECDSA.
	Questionable extension field!
  Oakley-EC2N-4: 
	IPSec/IKE/Oakley curve #4 over a 185 bit binary field.
	Not suitable for ECDSA.
	Questionable extension field!

Now generate new private key with chosen curve (prime256v1 looks fine, like: c2pnb272w1, sect283k1, sect283r1 or secp256k1, etc)

$ openssl ecparam -out ec_key.pem -name prime256v1 -genkey

And generate self-signed certificate that could be directly used:

$ openssl req -new -key ec_key.pem -x509 -nodes -days 365 -out cert.pem
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:PL
State or Province Name (full name) [Some-State]:example.pl  
Locality Name (eg, city) []:example.pl
Organization Name (eg, company) [Internet Widgits Pty Ltd]:example.pl
Organizational Unit Name (eg, section) []:example.pl
Common Name (e.g. server FQDN or YOUR name) []:example.pl
Email Address []:hostmaster@example.pl

Running Apache with mod_spdy and PHP-FPM

SPDY is new protocol proposed by Google as an alternative for HTTP(S). Currently Chrome and Firefox browsers are using it as default if available on server. It is faster in most cases by few to several percent. The side effect of using mod_spdy is that it’s working well only with thread safe Apache’s modules. PHP module for Apache is not thread safe so we need to use PHP as CGI or FastCGI service. CGI is slow – so running mod_spdy for performance gain with CGI is simply pointless. FastCGI is better but it’s not possible to share APC cache in FastCGI mode (ex. using spawn-fcgi), so it’s poor too. Best for PHP is PHP-FPM which is FastCGI service with dynamic process manager and could use full advantages of APC. In such configuration I could switch from apache prefork to worker which should use less resources and be more predictable.

Installation

On Squeeze we need to install dot.deb repository – instructions are here: http://www.dotdeb.org/instructions/

Then we could install:

apt-get install apache2-mpm-worker php5-fpm libapache2-mod-fastcgi

Now, mod_spdy – packages are available here: https://developers.google.com/speed/spdy/mod_spdy/ Choose your architecture.

wget https://dl-ssl.google.com/dl/linux/direct/mod-spdy-beta_current_i386.deb
dpkg -i mod-spdy-beta_current_i386.deb

Installation of this package will add automatically a new apt repository for mod_spdy.

If you have Apache’s module for PHP still installed you should remove it (you won’t need in anymore):

apt-get purge libapache2-mod-php5

Configuring PHP-FPM

First I’m changing php-fpm default pool configuration file – edit /etc/php5/fpm/pool.d/www.conf

; I want it to listen on socket, not on port
listen = /var/run/php5-fpm/site1.socket

;uncomment to set proper permission for socket
listen.owner = www-data
listen.group = www-data
listen.mode = 0660

;uncomment and change to - PHP leaks, so kill child after 100 requests
pm.max_requests = 100

; for proper chroot handling we will need also
php_admin_value[doc_root] = /var/www/site1
php_admin_value[cgi.fix_pathinfo] = 0

Now restart php-fpm:

service php5-fpm restart

Connecting Apache with PHP-FPM

In VirtualHost paste this:

<IfModule mod_fastcgi.c>
  Alias /php5.fcgi /var/www/site1/php5.fcgi
  FastCGIExternalServer /var/www/site1/php5.fcgi -socket /var/lib/apache2/fastcgi/site1.socket
  AddType application/x-httpd-fastphp5 .php
  Action application/x-httpd-fastphp5 /php5.fcgi

  <Directory "/var/www/site1/">
    Order deny,allow
    Deny from all
    <Files "php5.fcgi">
      Order allow,deny
      Allow from all
    </Files>
  </Directory>
</IfModule>

Enable needed modules and restart Apache:

a2enmod actions
a2enmod fastcgi
service apache2 restart

SSL

SPDY requires encrypted connection so you need configured SSL (virtualhost running on port 443). Typical configuration for SSL looks similar to this:


# some random stuff - exactly like in you NON SSL configuration :-)
SSLEngine on

SSLCertificateFile    /etc/ssl/certs/example.com.crt
SSLCertificateKeyFile /etc/ssl/private/example.com.priv.key
SSLCACertificateFile  /etc/ssl/private/ca.crt

Testing

Should work now 🙂
So, use Chromium, enter the site you just configured and then on second tab go to: chrome://net-internals/#spdy. You should see your site there if it’s running on SPDY.
You could also use plugins for Firefox or Chromium to test if site is running on SPDY.

Advertise SPDY on HTTP

When you test if SPDY is working fine (and is faster in your configuration) you could advertise availability of SPDY protocol on your HTTP VirtualHost. Thanks to that when browser supports SPDY it will use it for faster access. To do this just add header in configuration:

Header set Alternate-Protocol "443:spdy/2"

There are more options that could be used, if you need just check docs here.

Fortigate – VPN IPSec PSK XAuth z Android’a 4.x

Do niedawna na moim telefonie VPN’ami były: PPTP lub L2TP – oba niespecjalnie mi się podobały. Ale od wersji 4-tej pojawiły się dwa nowe tryby: IPSec Xauth PSK i IPSec Xauth RSA. W pierwszym autoryzacja wykorzystuje login i hasło, w drugim certyfikaty.

Tryb IPSec Xauth PSK jest bardzo wygodny bo łatwo można połączyć go z zewnętrznymi mechanizmami uwierzytelniającymi np. LDAP, Active Directory, itp.

Pokażę jak skonfigurować swojego Fortigate’a by umożliwić połączenie z telefonów i tabletów na Androidzie 4.x do “Intranetu”. Większość konfiguracji można przeprowadzić tylko w trybie CLI – zakładam że wiesz jak to zrobić. To co wygodniej można zrobić w trybie WWW to głównie tworzenie reguł dostępu na zaporze.

P.S. Teoretycznie powinno to też zadziałać na IPhone’ach/IPad’ach itp., przy czym udało mi się to zestawić na 4GS ale na 5-ce już nie – nie miałem tych aparatów na tyle długo by dokładniej to zbadać.

Na początek konfigurujemy fazę pierwszą – ja lubię interface mode ale oczywiście będzie to działać również w trybie policy (trzeba będzie nieco polecenia zmienić).

config vpn ipsec phase1-interface
  edit "vpn-android"
    set type dynamic
    set interface "port1"
    set dhgrp 2
    set peertype one
    set xauthtype auto
    set mode aggressive
    set mode-cfg enable
    set proposal aes128-sha1 aes128-md5 3des-sha1
    set negotiate-timeout 15
    set peerid "jakis-mobilny-identyfikator"
    set authusrgrp "grupa-ludzikow"
    set psksecret haslo
    set unity-support enable
    set ipv4-start-ip 172.16.0.5
    set ipv4-end-ip 172.16.0.100
    set ipv4-netmastk 255.255.255.0
    set ipv4-dns-server1 172.16.0.1
    set ipv4-dns-server2 8.8.8.8
  next
end

Jedna ciekawostka – opcja unity-support domyślnie jest ustawiana na enable i proponuję ją tak zostawić – dzięki temu ta sama konfiguracja będzie działać na urządzeniach z iOS’e i OS X, wystarczy skonfigurować VPN jako Cisco 🙂

No to faza druga:

config vpn ipsec phase2-interface
  edit "mobile-vpn-p2"
    set keepalive enable
    set pfs disable
    set phase1name "vpn-android"
    set proposal aes128-sha1 aes128-md5 3des-sha1
    set keylifeseconds 3600
  next
end

Kilka ustawień na interfejsie:

config system interface
  edit "vpn-android"
    set ip 172.16.0.1 255.255.255.255
    set allowaccess ping
    set type tunnel
    set remote-ip 139.x.x.10
    set interface "port1"
  next
end

Włączymy sobie serwer DNS na interfejsie VPN:

config system dns-server
  edit "vpn-android"
  next
end

No to teraz trzeba utworzyć co najmniej jedną regułę na zaporze zezwalającą na dostęp z interfejsu vpn-android (z adresów tej sieci) do jakiejś sieci intranetowej (np. wewnętrznego serwera WWW), na odpowiednich portach.

Ponadto jeśli chcemy móc korzystać z cache DNS’a na interfejsie vpn-android to musimy umożliwić do niego dostęp tworząc regułę z interfejsu vpn-android (z adresów przydzielanych przez mode-cfg) do vpn-android (przynajmniej do DNS’a – 172.16.0.1).

Jeżeli chcemy by urządzenia mobilne mogły łączyć się do zewnętrznych usług przez tunel VPN (co w niektórych przypadkach może być bardzo cenne) to musimy też utworzyć regułę z interfejsu vpn-android do któregoś z interfejsów WAN’owskich.

VPN Screenshot
A tak należy to wyklikać na telefonie

http://kb.fortinet.com/kb/viewContent.do?externalId=FD31619&sliceId=1