soyMAIL 2.1.0 requires JavaScript
soyMAIL @ wasd.vsm.com.au
       info-WASD Mailing List 2023 

Thu 19:47:14 Message "2023 / 0099" opened.  MIME.  utf-8.  1 attachment(s).  2 part(s).  145 kbytes.    JavaScript

Subject:[Info-WASD] HTTP/2 in the Real (WASD) World0099 / 0000
From:Mark.Daniel@wasd.vsm.com.au
Reply-to:info-wasd@vsm.com.au
Date:Tue, 24 Oct 2023 14:09:09 +1030  [24-OCT-2023 14:09]
To:info-WASD@vsm.com.au

TL;DR (HTTP/2 currently topical) 'the real world' of (a mere) three systems.

As VSI announced the rebirth of [Open]VMS in its licensing deal with HP
during 2014, WASD v10 was essentially under care and maintenance, a lot like
HP VMS then but with the added 'care'.  The further announcement of an X86-64
port breathed the potential of a completely new life for VMS.

WASD was no longer in mothballs and v11 eventually emerged supporting the
then recently standardised HTTP/2.

  https://en.wikipedia.org/wiki/HTTP/2
  https://wasd.vsm.com.au/wasd_root/wasdoc/features/#http2


HTTP/2 Pros/Cons
----------------
The author's opinion; HTTP/2 takes all the complexity of TCP and pushes
behaviour up from OSI layers 4 and 5, into layer 7, and then adds some more
(complexity) for good measure.  Just my AU$0.05 worth.

It does demonstrate some distinct performance benefits, particularly for
modern, small size request/responses typical of HTTP based applications
(possibly its entire intent).  For larger responses little benefit.  How does
it accomplish this?  Multiplexing requests over a single TCP connection. 
Binary, compressed request/response headers.  Prioritisation of some requests
over others.

It pushes processing overhead towards both client and server; memory and CPU.

It also pushes additional complexity onto the developer.  HTTP used to be a
simple protocol.  Easier debugging := easily readable.  Less of an issue when
using a library of functionality but as any WASD deployer knows the author
prefers to roll-his-own.  I really think HTTP/2 has cured him of that.


So what of the "Real World"?
----------------------------
The attachment HTTP.png shows the WASD HTTP Report extracted from three
Internet facing systems as a single image.  Each system has its own purpose
and workload, so provide a diverse testbed.  All are using the same version
of WASD.

As Internet facing systems generally experience large volumes of crawler
traffic, and observation has confirmed much of this is still HTTP/1.1 (only
recently a minority of crawlers using HTTP/2) it is unsurprising the traffic
is biased against HTTP/2.  Nevertheless we press on.

Two systems present a good case for HTTP/2.  One quite the converse.

All systems show comparable minimum and maximum response sizes.  The middle
one significantly smaller averages to the other two.  Knowing the activity
profile of that system this is not unexpected.

Network maximum transfer rates are proportionally comparable between /2 and
/1.1 across the three systems.  The average rates show some diversity, with
the middle one again not unexpected.  Two systems show significantly lower
HTTP/2 durations (request-response RTT), roughly 1/5x of HTTP/1.1 (~20%)
respectively, and one quite the converse at 7x HTTP/1.1.  Go figure.

YMMV (with workload profile).

  https://wasd.vsm.com.au/wasd_root/wasdoc/features/#http2andperformance


HTTP/2 Vulnerabilities
----------------------
A small number of potential (D)DoS vulnerabilities were noted in the original
RFC and then further emphasised by a Vulnerability Note in 2019.

  https://datatracker.ietf.org/doc/html/rfc7540#section-10.5
  https://www.kb.cert.org/vuls/id/605641/

WASD v11.5.0 introduced recommended mitigations.


HTTP/2 Rapid Reset Vulnerability
--------------------------------
A RECENTly announced additional vulnerability to (D)DoS attack, exploits
HTTP/2 ability to set up multiple requests within a single TCP connection --
and cancel them indeterminately.  This has been reported by the big players,
Google, Amazon, Microsoft, Cloudflare, as very effective, setting record
attacks from a much smaller than usual base of compromised/complicit systems.

  https://www.cvedetails.com/cve/CVE-2023-44487
  https://www.bleepingcomputer.com/news/security/new-http-2-rapid-reset-zero-day-attack-breaks-ddos-records/
  https://github.com/bcdannyboy/CVE-2023-44487 **

** Note: this is a basic test for site HTTP/2, not a site DoS stress-test.

Because WASD supports HTTP/2 there is the possibility of attack.
Although the Python checking code did not report VULNERABLE or LIKELY.

|mark@MacMini1 ~ % cat input_urls.txt
|https://192.168.1.3/
|https://192.168.1.86/
|mark@MacMini1 ~ % python3 cve202344487.py -i input_urls.txt -o output_results.csv
|mark@MacMini1 ~ % cat output_results.csv
|Timestamp,Source Internal IP,Source External IP,URL,Vulnerability Status,Error/Downgrade Version
|2023-10-20 09:46:08,192.168.1.2,...,https://192.168.1.3/,POSSIBLE,Failed to send RST_STREAM: send_rst_stream_h2 - 3
|2023-10-20 09:46:10,192.168.1.2,...,https://192.168.1.86/,POSSIBLE,Failed to send RST_STREAM: send_rst_stream_h2 - 3

As reported by the test code cve202344487.py it fails to get to send the
RST_STREAM.  In this test code that seems only to occur after the request
header is sent and WASD has returned the response.  Modifying the test code
to send the RST_STREAM immediately following the request header makes the
RST_STREAM visible and so WASD (in common with HTTP/2 servers in general)
would be vulnerable to such an attack (it is just not revealed by the as
published test code).

|13:19:43.47 HTTP2WAT 0166 001004 HTTP/2     FRAME client length:4 type:3 RST_STREAM flags:0x00 ident:1|
|13:19:43.47 HTTP2WAT 0430 001004 HTTP/2     RST_STREAM ident:1 error:0 "graceful shutdown"|

WASD already contains a mitigation for a related vulnerability

  https://www.cvedetails.com/cve/CVE-2019-9514/

that would go a long way to ameliorating a Rapid Reset attack.  It will be
refined and also used to mitigate CVE-2023-44487.

Though VMS systems using WASD seem unlikely to be attacked, v12.2 (to be
released Real Soon Now -- just waiting on a refreshed X86 C compiler) will
include these refinements.

Should you conclude your site is being HTTP/2 (D)DoSed merely

  |#WASD_CONFIG_GLOBAL
  [Http2StreamsMax] 10

or more drastically

  |#WASD_CONFIG_GLOBAL
  [Http2Protocol] disable

and HTTPD/DO=RESTART=NOW.  The typical VMS service won't lose much at all.

This item is one of a collection at
https://wasd.vsm.com.au/other/#occasional

  ¤¤¤       
  ¤¤¤     
  ¤¤¤     
Image: 1st click 100%, 2nd actual size, 3rd default again