CGIplus-enabled Run-time Environment Example -------------------------------------------- ***** FIRST, EVIDENCE OF PERSISTANCE ***** Usage Count: 1 ***** SECOND, THE CGI ENVIRONMENT AVAILABLE ***** WWW_AUTH_TYPE= WWW_CONTENT_LENGTH=0 WWW_CONTENT_TYPE= WWW_CSP_NONCE=c55a00522fc02ac9655afad7b0e5e8e WWW_DOCUMENT_ROOT= WWW_GATEWAY_INTERFACE=CGI/1.1 WWW_GATEWAY_EOF=$Z-1A73E2B1F40084BEEFE5F227- WWW_GATEWAY_EOT=$D-BA95E43CEADAD00DFDD6436A- WWW_GATEWAY_ESC=$E-696A5DE32ED4B5C80358B929- WWW_GATEWAY_MRS=16492 WWW_HTTP_ACCEPT=*/* WWW_HTTP_USER_AGENT=Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com) WWW_HTTP_ACCEPT_ENCODING=gzip, br, zstd, deflate WWW_HTTP_HOST=wasd.vsm.com.au WWW_PATH_INFO= WWW_PATH_TRANSLATED= WWW_QUERY_STRING= WWW_REMOTE_ADDR=3.143.247.50 WWW_REMOTE_HOST=ec2-3-143-247-50.us-east-2.compute.amazonaws.com WWW_REMOTE_PORT=2360 WWW_REMOTE_USER= WWW_REQUEST_METHOD=GET WWW_REQUEST_PROTOCOL=HTTP/1.1 WWW_REQUEST_SCHEME=http: WWW_REQUEST_TIME_GMT=Fri, 01 Nov 2024 02:33:15 GMT WWW_REQUEST_TIME_LOCAL=Fri, 01 Nov 2024 13:03:15 WWW_REQUEST_URI=/rtbin/dict.c WWW_SCRIPT_FILENAME=WASD_ROOT:[src.httpd]dict.c WWW_SCRIPT_NAME=/rtbin/dict.c WWW_SCRIPT_RTE=cgi-bin:[000000]rte_example.exe WWW_SERVER_ADDR=119.252.17.13 WWW_SERVER_CHARSET=ISO-8859-1 WWW_SERVER_GMT=+10:30 WWW_SERVER_NAME=wasd.vsm.com.au WWW_SERVER_PROTOCOL=HTTP/1.1 WWW_SERVER_PORT=80 WWW_SERVER_SIGNATURE=
WASD/12.2.5 Server at wasd.vsm.com.au Port 80
WWW_SERVER_SOFTWARE=HTTPd-WASD/12.2.5 OpenVMS/IA64 SSL WWW_UNIQUE_ID=70c3bc4dcd11d40ce59 WWW_KEY_COUNT=0 ***** THIRD, AN "INTERPRETED" FILE (WWW_SCRIPT_NAME/WWW_SCRIPT_FILENAME) ***** [0001] /*****************************************************************************/ [0002] /* [0003] dict.c [0004] [0005] A hash dictionary (associative array) of key-value pairs. Both key and value [0006] are commonly strings but actually can be any sized blob. Keys that begin with [0007] a digit have special meaning to response header processing and so should be [0008] avoided in that type (especially). Inserting a key that already exists [0009] effectively replaces the previous value - unless - the uniquifier bit :-} is [0010] set, in which case a dictionary-unique integer (followed by a colon) is [0011] prefixed to the key allowing multiple entries having (essentially) the same [0012] key. As memory allocation can change due to value length increase any [0013] reference to the entry or its value must be re-set. [0014] [0015] All keys are pushed to lower-case on insertion, though key lookup is case [0016] insensitive. Each key has a "type", represented by a specific, [0017] non-alphanumeric character. [0018] [0019] o $ request processing (server internal) [0020] o ~ meta configuration (DICT and SET DICT=..) [0021] o > request header field [0022] o < response header field [0023] [0024] (There are others, some currently unused.) These allow entries that can be [0025] discriminated between quite simply. For example, iterating through all request [0026] header fields. Some configuration uses of the dictionary (i.e. the DICT [0027] directive and the mapping SET DICT=.. rule) will accept a key with a leading [0028] non-alpha-numeric that allows a specific type key to be looked up and also [0029] inserted. This allows the perhaps dangerous changing of server internal, [0030] request processing dictionary entries. But hey, sysadmins know what they're [0031] doing :-} [0032] [0033] A load factor of 10 collision list entries to 1 hash table element [0034] (empirically) seems to be a performance sweet spot so a table size of 32 means [0035] up to some 300 entries will be efficiently handled (and theoretically, a table [0036] of 512 elements would efficiently support 50k entries - though that's not the [0037] design objective for this purpose). For server request processing purposes the [0038] table size is set at 32 although expected entries would number less than a [0039] couple of dozen (request and response header fields predominating). [0040] [0041] A dictionary built using request memory does not need to be explicitly [0042] destroyed when RequestFreeHeap() is deployed. If the request is just zeroized [0043] then it does need destruction ready for any subsequent request. [0044] [0045] With HTTP/1.n and HTTP/2 having fundamentally different header representations [0046] there was a distinct need for an intermediate abstraction from which these both [0047] could be interfaced with the server-internal requirements. Although the [0048] implementation is quite general purpose, the role of this WASD dictionary is to [0049] act as a repository for request and response header field names and values, [0050] internally used strings (and potentially binary objects), and a new [0051] configuration and request processing (mapping/authorization) abstraction based [0052] on matching key-value pairs. (And it gave me a chance to play with an [0053] associative array implementation :-) [0054] [0055] The implementation is a bog-standard hash array with collision linked-list. An [0056] associated list allows iteration in order of insertion. Each associative array [0057] (dictionary) is dynamically allocated, as are entries in those arrays. As the [0058] entry behaviour seems mostly static after insertion (i.e. values do not change [0059] often if at all) each entry is allocated with space enough for the entry data [0060] structure, key and value. One allocation per insertion. Makes value length [0061] increases a little more complex but that's the tradeoff. Memory is generally [0062] allocated from request heap zones and so disposes of itself when the request [0063] structure is itself disposed of. [0064] [0065] [0066] VERSION HISTORY [0067] --------------- [0068] 13-JAN-2020 MGD bugfix; "tmptr && tmptr->clink.." [0069] 25-FEB-2018 MGD 'uniquifier' delimiter changed from period to colon [0070] 22-DEC-2015 MGD initial [0071] */ [0072] /*****************************************************************************/ [0073] [0074] #ifdef WASD_VMS_V7 [0075] #undef _VMS__V6__SOURCE [0076] #define _VMS__V6__SOURCE [0077] #undef __VMS_VER [0078] #define __VMS_VER 70000000 [0079] #undef __CRTL_VER [0080] #define __CRTL_VER 70000000 [0081] #endif [0082] [0083] #include [0084] #include [0085] [0086] #include "wasd.h" [0087] [0088] #define WASD_MODULE "DICT" [0089] [0090] /******************/ [0091] /* global storage */ [0092] /******************/ [0093] [0094] static uint DictLookupHash, [0095] DictLookupKlen; [0096] [0097] /********************/ [0098] /* external storage */ [0099] /********************/ [0100] [0101] extern char ErrorSanityCheck []; [0102] [0103] extern CONFIG_STRUCT Config; [0104] extern WATCH_STRUCT Watch; [0105] [0106] /*****************************************************************************/ [0107] /* [0108] Create a new dictionary. [0109] */ [0110] [0111] DICT_STRUCT* DictCreate [0112] ( [0113] REQUEST_STRUCT *rqptr, [0114] int size [0115] ) [0116] { [0117] int idx; [0118] DICT_STRUCT *dicptr; [0119] [0120] /*********/ [0121] /* begin */ [0122] /*********/ [0123] [0124] if (WATCHMOD (rqptr, WATCH_MOD__OTHER)) [0125] WatchThis (WATCHITM(rqptr), WATCH_MOD__OTHER, "DictCreate()"); [0126] [0127] if (rqptr) [0128] dicptr = VmGetHeap (rqptr, sizeof(DICT_STRUCT)); [0129] else [0130] dicptr = VmGet (sizeof(DICT_STRUCT)); [0131] dicptr->RequestPtr = rqptr; [0132] [0133] if ((int)(dicptr->size = size) <= 0) dicptr->size = DICT_DEFAULT_SIZE; [0134] if (dicptr->size > 4096) [0135] ErrorExitVmsStatus (SS$_BUGCHECK, ErrorSanityCheck, FI_LI); [0136] dicptr->bytes = sizeof(DICT_ENTRY_STRUCT*) * dicptr->size; [0137] [0138] if (rqptr) [0139] dicptr->table = VmGetHeap (rqptr, dicptr->bytes); [0140] else [0141] dicptr->table = VmGet (dicptr->bytes); [0142] [0143] return (dicptr); [0144] } [0145] [0146] /*****************************************************************************/ [0147] /* [0148] Delete all dictionary items and then the dictionary itself. [0149] Returns a NULL pointer. [0150] */ [0151] [0152] DICT_STRUCT* DictDestroy (DICT_STRUCT *dicptr) [0153] [0154] { [0155] DICT_ENTRY_STRUCT *denptr, *flink; [0156] REQUEST_STRUCT *rqptr; [0157] [0158] /*********/ [0159] /* begin */ [0160] /*********/ [0161] [0162] rqptr = dicptr->RequestPtr; [0163] [0164] if (WATCHMOD (rqptr, WATCH_MOD__OTHER)) [0165] WatchThis (WATCHITM(rqptr), WATCH_MOD__OTHER, "DictDestroy()"); [0166] [0167] for (denptr = dicptr->head; denptr != NULL; denptr = flink) [0168] { [0169] flink = denptr->flink; [0170] if (rqptr) [0171] VmFreeFromHeap (rqptr, denptr, FI_LI); [0172] else [0173] VmFree (denptr, FI_LI); [0174] } [0175] [0176] if (rqptr) [0177] VmFreeFromHeap (rqptr, dicptr, FI_LI); [0178] else [0179] VmFree (dicptr, FI_LI); [0180] [0181] return (NULL); [0182] } [0183] [0184] /*****************************************************************************/ [0185] /* [0186] Return the value of the dirty flag and reset. [0187] */ [0188] [0189] uint DictDirty (DICT_STRUCT *dicptr) [0190] [0191] { [0192] uint dirty; [0193] [0194] /*********/ [0195] /* begin */ [0196] /*********/ [0197] [0198] if (dicptr == NULL) return (NULL); [0199] [0200] dirty = dicptr->dirty; [0201] dicptr->dirty = 0; [0202] return (dirty); [0203] } [0204] [0205] /*****************************************************************************/ [0206] /* [0207] Return a pointer to an exact duplicate of the supplied dictionary. If the [0208] request pointer is supplied it is duplicated into that request. This can be [0209] the same request as the dictionary being duplicated or another. If the [0210] request pointer is NULL then it is duplicated into global memory, and of [0211] course that may be where the original dictionary also resides. It can be [0212] duplicated in any combination. [0213] */ [0214] [0215] DICT_STRUCT* DictDuplicate [0216] ( [0217] REQUEST_STRUCT *rqptr, [0218] DICT_STRUCT *dicptr [0219] ) [0220] { [0221] DICT_STRUCT *duptr; [0222] DICT_ENTRY_STRUCT *denptr, *nxtptr; [0223] [0224] /*********/ [0225] /* begin */ [0226] /*********/ [0227] [0228] if (dicptr == NULL) return (NULL); [0229] [0230] duptr = DictCreate (rqptr, dicptr->size); [0231] nxtptr = dicptr->head; [0232] while ((denptr = nxtptr) != NULL) [0233] { [0234] nxtptr = denptr->flink; [0235] DictInsert (duptr, denptr->type, denptr->key, denptr->klen, [0236] denptr->value, denptr->vlen); [0237] } [0238] return (duptr); [0239] } [0240] [0241] /*****************************************************************************/ [0242] /* [0243] Expand the specified entry's available value space by the specified quantity, [0244] copying any existing content to the new memory. As the memory allocation is [0245] likely to change any reference to the entry or its value must be re-set. [0246] */ [0247] [0248] DICT_ENTRY_STRUCT* DictExpandEntry [0249] ( [0250] DICT_ENTRY_STRUCT *denptr, [0251] uint bytes [0252] ) [0253] { [0254] ulong hash; [0255] uchar *cptr, *sptr, *zptr; [0256] DICT_STRUCT *dicptr; [0257] DICT_ENTRY_STRUCT *newptr, *tmptr; [0258] REQUEST_STRUCT *rqptr; [0259] [0260] /*********/ [0261] /* begin */ [0262] /*********/ [0263] [0264] dicptr = denptr->DictPtr; [0265] rqptr = dicptr->RequestPtr; [0266] [0267] if (WATCHMOD (rqptr, WATCH_MOD__OTHER)) [0268] WatchThis (WATCHITM(rqptr), WATCH_MOD__OTHER, "DictExpandEntry()"); [0269] [0270] /* this is *additional* value space */ [0271] bytes += denptr->bytes; [0272] [0273] /* plus two terminating nulls and some "elbow-room" */ [0274] if (rqptr) [0275] newptr = VmGetHeap (rqptr, sizeof(DICT_ENTRY_STRUCT) + bytes + 8); [0276] else [0277] newptr = VmGet (sizeof(DICT_ENTRY_STRUCT) + bytes + 8); [0278] [0279] newptr->bytes = bytes; [0280] newptr->hash = hash = denptr->hash; [0281] newptr->type[0] = denptr->type[0]; [0282] newptr->DictPtr = denptr->DictPtr; [0283] [0284] /* reproduce the key */ [0285] newptr->klen = denptr->klen; [0286] zptr = (sptr = newptr->key = newptr->buffer) + newptr->klen; [0287] for (cptr = denptr->key; sptr < zptr; *sptr++ = *cptr++); [0288] [0289] /* reproduce any value (see DictInsert() "just a little too clever") */ [0290] newptr->value = newptr->buffer + newptr->klen + 1; [0291] /* if just reserving space (still valid with an expansion) */ [0292] if (denptr->value == NULL) [0293] newptr->vlen = 0; [0294] else [0295] { [0296] zptr = (sptr = newptr->value) + (newptr->vlen = denptr->vlen); [0297] for (cptr = denptr->value; sptr < zptr; *sptr++ = *cptr++); [0298] } [0299] [0300] /* adjust the hash table / collision list */ [0301] if (dicptr->table[hash] == denptr) [0302] dicptr->table[hash] = newptr; [0303] else [0304] { [0305] for (tmptr = dicptr->table[hash]; [0306] tmptr && tmptr->clink != denptr; [0307] tmptr = tmptr->clink); [0308] tmptr->clink = newptr; [0309] } [0310] newptr->clink = denptr->clink; [0311] [0312] /* adjust the ordered list */ [0313] if (dicptr->head == denptr) dicptr->head = newptr; [0314] if (dicptr->tail == denptr) dicptr->tail = newptr; [0315] if (denptr->flink != NULL) denptr->flink->blink = newptr; [0316] if (denptr->blink != NULL) denptr->blink->flink = newptr; [0317] newptr->flink = denptr->flink; [0318] newptr->blink = denptr->blink; [0319] [0320] /* if necessary adjust the iterator */ [0321] if (dicptr->next == denptr) dicptr->next = newptr; [0322] [0323] if (rqptr) [0324] VmFreeFromHeap (rqptr, denptr, FI_LI); [0325] else [0326] VmFree (denptr, FI_LI); [0327] [0328] return (newptr); [0329] } [0330] [0331] /*****************************************************************************/ [0332] /* [0333] Extract the specified entry from its dictionary. [0334] */ [0335] [0336] DICT_ENTRY_STRUCT* DictExtractEntry (DICT_ENTRY_STRUCT *denptr) [0337] [0338] { [0339] uint hash; [0340] DICT_STRUCT *dicptr; [0341] DICT_ENTRY_STRUCT *tmptr; [0342] REQUEST_STRUCT *rqptr; [0343] [0344] /*********/ [0345] /* begin */ [0346] /*********/ [0347] [0348] dicptr = denptr->DictPtr; [0349] rqptr = dicptr->RequestPtr; [0350] [0351] if (WATCHMOD (rqptr, WATCH_MOD__OTHER)) [0352] WatchThis (WATCHITM(rqptr), WATCH_MOD__OTHER, "DictExtractEntry()"); [0353] [0354] /* extract from the hash table / collision list */ [0355] hash = denptr->hash; [0356] if (dicptr->table[hash] == denptr) [0357] dicptr->table[hash] = denptr->clink; [0358] else [0359] { [0360] for (tmptr = dicptr->table[hash]; [0361] tmptr && tmptr->clink != denptr; [0362] tmptr = tmptr->clink); [0363] tmptr->clink = denptr->clink; [0364] } [0365] [0366] /* remove from the ordered list */ [0367] if (dicptr->head == denptr) dicptr->head = denptr->flink; [0368] if (dicptr->tail == denptr) dicptr->tail = denptr->blink; [0369] if (denptr->flink != NULL) denptr->flink->blink = denptr->blink; [0370] if (denptr->blink != NULL) denptr->blink->flink = denptr->flink; [0371] [0372] /* if necessary adjust the iterator */ [0373] if (dicptr->next == denptr) dicptr->next = denptr->flink; [0374] [0375] dicptr->count--; [0376] dicptr->bytes -= denptr->bytes; [0377] dicptr->dirty++; [0378] [0379] /* for detecting any subsequent (re)insertion */ [0380] denptr->extract = dicptr->size; [0381] [0382] return (denptr); [0383] } [0384] [0385] /*****************************************************************************/ [0386] /* [0387] Insert the specified key and value into the specified dictionary. If the key [0388] already exists then that entry will be removed and replaced with this one. If [0389] the type has the DICT_TYPE_UNIQUE bit set then unique entries are created by [0390] prefixing the key with a number 1..n and period (e.g. "10:example"). Generally [0391] the digits and colon leading the supplied key need to be excised before use. [0392] */ [0393] [0394] DICT_ENTRY_STRUCT* DictInsert [0395] ( [0396] DICT_STRUCT *dicptr, [0397] uchar *type, [0398] uchar *key, [0399] int klen, [0400] uchar *value, [0401] int vlen [0402] ) [0403] { [0404] static char type2 [2]; [0405] static char KeyBuf [256]; [0406] static $DESCRIPTOR (KeyBufDsc, KeyBuf); [0407] static $DESCRIPTOR (KeyFaoDsc, "!UL:!#AZ"); [0408] [0409] BOOL reuse; [0410] int cnt, len; [0411] ushort slen; [0412] ulong bytes, hash; [0413] uchar *cptr, *sptr, *zptr; [0414] DICT_ENTRY_STRUCT *denptr, *tmptr; [0415] REQUEST_STRUCT *rqptr; [0416] [0417] /*********/ [0418] /* begin */ [0419] /*********/ [0420] [0421] rqptr = dicptr->RequestPtr; [0422] [0423] if (WATCHMOD (rqptr, WATCH_MOD__OTHER)) [0424] WatchThis (WATCHITM(rqptr), WATCH_MOD__OTHER, [0425] "DictInsert() !&C !SL !#AZ = !SL !#AZ", [0426] type[0], klen, klen, key, vlen, vlen, value); [0427] [0428] /* cannot use an empty key */ [0429] if (klen == 0 || !key[0]) return (NULL); [0430] [0431] if (vlen < 0) [0432] { [0433] for (cptr = value; *cptr; cptr++); [0434] vlen = cptr - value; [0435] } [0436] [0437] /* buffer a type that's free of any potential "uniquifier bit" */ [0438] type2[0] = type[0] & ~DICT_TYPE_UNIQUE; [0439] [0440] denptr = DictLookup (dicptr, type2, key, klen); [0441] [0442] /* cannot use a silly (buggy?) sized key (allowing for enumeration) */ [0443] if ((klen = DictLookupKlen) > sizeof(KeyBuf)-8) return (NULL); [0444] [0445] /* generated by DictLookup() no need to do it again */ [0446] hash = DictLookupHash; [0447] [0448] if (denptr != NULL) [0449] { [0450] /* entry already exists */ [0451] if ((type[0] & DICT_TYPE_UNIQUE) && [0452] type2[0] != DICT_TYPE_OPAQUE_KEY) [0453] { [0454] /* enumerated (unique) entries */ [0455] sys$fao (&KeyFaoDsc, &slen, &KeyBufDsc, ++dicptr->unique, klen, key); [0456] KeyBuf[slen] = '\0'; [0457] key = KeyBuf; [0458] klen = slen; [0459] /* compute the hash */ [0460] hash = type2[0]; [0461] for (cptr = key; slen--; cptr++) hash = hash * DICT_HASH_MULT + *cptr; [0462] hash = hash % dicptr->size; [0463] /* need a fresh one */ [0464] denptr = NULL; [0465] reuse = false; [0466] } [0467] else [0468] { [0469] /* can the entry's value buffer be reused */ [0470] if (klen + vlen > denptr->bytes) [0471] { [0472] /* won't fit, expand using new value (just a little too clever) */ [0473] denptr->value = value; [0474] denptr->vlen = vlen; [0475] denptr = DictExpandEntry (denptr, (klen + vlen) - denptr->bytes); [0476] return (denptr); [0477] } [0478] /* existing entry can contain the new value */ [0479] reuse = true; [0480] } [0481] } [0482] [0483] /* if entry did not already exist allocate and initialise a fresh one */ [0484] if (denptr == NULL) [0485] { [0486] bytes = klen + vlen; [0487] /* to (perhaps) improve memory management impose a minimum sized chunk */ [0488] if (bytes < DICT_MIN_BYTES) bytes = DICT_MIN_BYTES; [0489] [0490] /* plus two terminating nulls and some "elbow-room" */ [0491] if (rqptr) [0492] denptr = VmGetHeap (rqptr, sizeof(DICT_ENTRY_STRUCT) + bytes + 8); [0493] else [0494] denptr = VmGet (sizeof(DICT_ENTRY_STRUCT) + bytes + 8); [0495] [0496] denptr->bytes = bytes; [0497] denptr->hash = hash; [0498] denptr->type[0] = type2[0]; [0499] denptr->DictPtr = dicptr; [0500] [0501] denptr->klen = klen; [0502] zptr = (sptr = denptr->key = denptr->buffer) + klen; [0503] /* push key to lower-case! (no null required in zeroed memory) */ [0504] for (cptr = key; sptr < zptr; *sptr++ = to_lower(*cptr++)); [0505] reuse = false; [0506] } [0507] [0508] denptr->value = denptr->buffer + denptr->klen + 1; [0509] /* if the value is NULL then just reserving space */ [0510] if (value == NULL) [0511] denptr->vlen = 0; [0512] else [0513] { [0514] zptr = (sptr = denptr->value) + (denptr->vlen = vlen); [0515] for (cptr = value; sptr < zptr; *sptr++ = *cptr++); [0516] /* in case a shorter value is being overwritten */ [0517] *sptr = '\0'; [0518] } [0519] [0520] if (reuse) [0521] dicptr->dirty++; [0522] else [0523] DictInsertEntry (dicptr, denptr); [0524] [0525] return (denptr); [0526] } [0527] [0528] /*****************************************************************************/ [0529] /* [0530] Insert the specified entry into the specified dictionary. [0531] */ [0532] [0533] DICT_ENTRY_STRUCT* DictInsertEntry [0534] ( [0535] DICT_STRUCT *dicptr, [0536] DICT_ENTRY_STRUCT *denptr [0537] ) [0538] { [0539] int len; [0540] ulong hash; [0541] uchar *cptr; [0542] DICT_ENTRY_STRUCT *tmptr; [0543] REQUEST_STRUCT *rqptr; [0544] [0545] /*********/ [0546] /* begin */ [0547] /*********/ [0548] [0549] rqptr = dicptr->RequestPtr; [0550] [0551] if (WATCHMOD (rqptr, WATCH_MOD__OTHER)) [0552] WatchThis (WATCHITM(rqptr), WATCH_MOD__OTHER, "DictInsertEntry()"); [0553] [0554] /* if extracted from a(nother) dictionary */ [0555] if (denptr->extract) [0556] { [0557] /* check that the memory belongs to the same environment */ [0558] if (dicptr->RequestPtr != denptr->DictPtr->RequestPtr) [0559] ErrorExitVmsStatus (SS$_BUGCHECK, ErrorSanityCheck, FI_LI); [0560] [0561] /* check if there's an existing entry and remove as required */ [0562] if (tmptr = DictLookup (dicptr, denptr->type, denptr->key, denptr->klen)) [0563] DictRemoveEntry (tmptr); [0564] [0565] /* we've certainly done this! */ [0566] denptr->DictPtr = dicptr; [0567] [0568] if (denptr->extract != dicptr->size); [0569] { [0570] /* need to recalculate the hash */ [0571] hash = denptr->type[0]; [0572] len = denptr->klen; [0573] for (cptr = denptr->key; len--; cptr++) [0574] hash = hash * DICT_HASH_MULT + *cptr; [0575] denptr->hash = hash % dicptr->size; [0576] } [0577] [0578] /* reset the extraction indicator (size of original dictionary) */ [0579] denptr->extract = 0; [0580] } [0581] [0582] /* insert into the hash table / collision list */ [0583] hash = denptr->hash; [0584] if (dicptr->table[hash] == NULL) [0585] dicptr->table[hash] = denptr; [0586] else [0587] { [0588] for (tmptr = dicptr->table[hash]; [0589] tmptr && tmptr->clink != NULL; [0590] tmptr = tmptr->clink); [0591] tmptr->clink = denptr; [0592] } [0593] /* in case it's been extracted from another dictionary */ [0594] denptr->clink = NULL; [0595] [0596] /* append to the tail of the ordered list */ [0597] if (dicptr->head == NULL) dicptr->head = denptr; [0598] if (dicptr->tail != NULL) [0599] { [0600] dicptr->tail->flink = denptr; [0601] denptr->blink = dicptr->tail; [0602] } [0603] dicptr->tail = denptr; [0604] /* in case it's been extracted from another dictionary */ [0605] denptr->flink = NULL; [0606] [0607] dicptr->dirty++; [0608] dicptr->count++; [0609] dicptr->bytes += sizeof(DICT_ENTRY_STRUCT) + denptr->bytes; [0610] [0611] return (denptr); [0612] } [0613] [0614] /*****************************************************************************/ [0615] /* [0616] Iterate through dictionary entries. List is ordered first to last insertion. [0617] Second parameter as NULL resets to head, "*" selects all, "\0" those of [0618] that type, and "\0" a string match. Return a pointer to the entry [0619] or NULL on exhaustion or empty dictionary. On reset the returned pointer [0620] should only be considered informational as to empty or not. [0621] */ [0622] [0623] DICT_ENTRY_STRUCT* DictIterate [0624] ( [0625] DICT_STRUCT *dicptr, [0626] uchar *which [0627] ) [0628] { [0629] DICT_ENTRY_STRUCT *denptr; [0630] [0631] /*********/ [0632] /* begin */ [0633] /*********/ [0634] [0635] if (which != NULL) [0636] { [0637] while ((denptr = dicptr->next) != NULL) [0638] { [0639] dicptr->next = denptr->flink; [0640] if (!which[0] || MATCH2 (which, "*")) break; [0641] if (which[0] && !which[1]) [0642] if (denptr->type[0] == which[0]) [0643] break; [0644] else [0645] continue; [0646] if (StringMatch (NULL, denptr->key, which)) break; [0647] } [0648] return (denptr); [0649] } [0650] [0651] /* reset to start */ [0652] return (dicptr->next = dicptr->head); [0653] } [0654] [0655] /*****************************************************************************/ [0656] /* [0657] Look in the specified dictionary for the specified key. If key length is -1 [0658] then assume key is null-terminated string, otherwise some other opaque blob. [0659] String key matching is case-insensitive. Lookup includes the entry type. [0660] Return a pointer to the dictionary item found or NULL if not. [0661] */ [0662] [0663] DICT_ENTRY_STRUCT* DictLookup [0664] ( [0665] DICT_STRUCT *dicptr, [0666] uchar *type, [0667] uchar *key, [0668] int klen [0669] ) [0670] { [0671] uint len; [0672] ulong hash; [0673] uchar *cptr, *sptr, *zptr; [0674] DICT_ENTRY_STRUCT *denptr; [0675] REQUEST_STRUCT *rqptr; [0676] [0677] /*********/ [0678] /* begin */ [0679] /*********/ [0680] [0681] rqptr = dicptr->RequestPtr; [0682] [0683] if (WATCHMOD (rqptr, WATCH_MOD__OTHER)) [0684] WatchThis (WATCHITM(rqptr), WATCH_MOD__OTHER, [0685] "DictLookup() !&C !SL !#AZ", type[0], klen, klen, key); [0686] [0687] /* cannot use an empty key */ [0688] if (klen == 0 || (klen < 0 && !key[0])) return (NULL); [0689] [0690] /* make the type character part of the hash */ [0691] hash = type[0]; [0692] if (klen < 0) [0693] { [0694] for (cptr = key; *cptr; cptr++) hash = hash * DICT_HASH_MULT + *cptr; [0695] klen = cptr - key; [0696] } [0697] else [0698] { [0699] len = klen; [0700] for (cptr = key; len--; cptr++) hash = hash * DICT_HASH_MULT + *cptr; [0701] } [0702] hash = hash % dicptr->size; [0703] [0704] /* save these so that they don't need to be generated again */ [0705] DictLookupHash = hash; [0706] DictLookupKlen = klen; [0707] [0708] zptr = key + klen; [0709] for (denptr = dicptr->table[hash]; denptr != NULL; denptr = denptr->clink) [0710] { [0711] if (denptr->type[0] != type[0]) continue; [0712] if (denptr->klen != klen) continue; [0713] cptr = key; [0714] sptr = denptr->key; [0715] if (denptr->type[0] == DICT_TYPE_OPAQUE_KEY) [0716] while (cptr < zptr && *cptr == *sptr) { cptr++; sptr++; } [0717] else [0718] /* all keys are pushed to lower-case on insertion */ [0719] while (cptr < zptr && to_lower(*cptr) == *sptr) { cptr++; sptr++; } [0720] if (cptr == zptr && sptr == denptr->key + denptr->klen) break; [0721] } [0722] [0723] if (WATCHMOD (rqptr, WATCH_MOD__OTHER)) [0724] WatchThis (WATCHITM(rqptr), WATCH_MOD__OTHER, "denptr:!&X", denptr); [0725] [0726] return (denptr); [0727] } [0728] [0729] /*****************************************************************************/ [0730] /* [0731] Delete the specified key from the dictionary. Returns NULL if the key was not [0732] found or a pointer to the entry if the key was found. The pointer is [0733] informational and must not subsequently be accessed. [0734] */ [0735] [0736] DICT_ENTRY_STRUCT* DictRemove [0737] ( [0738] DICT_STRUCT *dicptr, [0739] uchar *type, [0740] uchar *key, [0741] int klen [0742] ) [0743] { [0744] uint hash; [0745] DICT_ENTRY_STRUCT *denptr; [0746] REQUEST_STRUCT *rqptr; [0747] [0748] /*********/ [0749] /* begin */ [0750] /*********/ [0751] [0752] rqptr = dicptr->RequestPtr; [0753] [0754] if (WATCHMOD (rqptr, WATCH_MOD__OTHER)) [0755] WatchThis (WATCHITM(rqptr), WATCH_MOD__OTHER, [0756] "DictRemove() !AZ !&Z", type, key); [0757] [0758] if ((denptr = DictLookup (dicptr, type, key, klen)) == NULL) return (NULL); [0759] [0760] DictExtractEntry (denptr); [0761] [0762] if (rqptr) [0763] VmFreeFromHeap (rqptr, denptr, FI_LI); [0764] else [0765] VmFree (denptr, FI_LI); [0766] [0767] return (denptr); [0768] } [0769] [0770] /*****************************************************************************/ [0771] /* [0772] Delete the specified entry from the dictionary. Returns the freed pointer, [0773] which is informational and must not subsequently be accessed. [0774] */ [0775] [0776] DICT_ENTRY_STRUCT* DictRemoveEntry (DICT_ENTRY_STRUCT *denptr) [0777] [0778] { [0779] DICT_STRUCT *dicptr; [0780] REQUEST_STRUCT *rqptr; [0781] [0782] /*********/ [0783] /* begin */ [0784] /*********/ [0785] [0786] dicptr = denptr->DictPtr; [0787] rqptr = dicptr->RequestPtr; [0788] [0789] if (WATCHMOD (rqptr, WATCH_MOD__OTHER)) [0790] WatchThis (WATCHITM(rqptr), WATCH_MOD__OTHER, "DictRemoveEntry()"); [0791] [0792] DictExtractEntry (denptr); [0793] [0794] if (rqptr) [0795] VmFreeFromHeap (rqptr, denptr, FI_LI); [0796] else [0797] VmFree (denptr, FI_LI); [0798] [0799] return (denptr); [0800] } [0801] [0802] /*****************************************************************************/ [0803] /* [0804] Set the value length of the specified entry. Check the size is within the [0805] storage allocated. Adjustments to the value string can be made (within [0806] constraints) because it's stored at the trailing end of the buffer space [0807] following the key. [0808] */ [0809] [0810] void DictValueLength [0811] ( [0812] DICT_ENTRY_STRUCT *denptr, [0813] uint length [0814] ) [0815] { [0816] DICT_STRUCT *dicptr; [0817] REQUEST_STRUCT *rqptr; [0818] [0819] /*********/ [0820] /* begin */ [0821] /*********/ [0822] [0823] dicptr = denptr->DictPtr; [0824] rqptr = dicptr->RequestPtr; [0825] [0826] if (WATCHMOD (rqptr, WATCH_MOD__OTHER)) [0827] WatchThis (WATCHITM(rqptr), WATCH_MOD__OTHER, "DictValueLength()"); [0828] [0829] if (denptr->klen + length > denptr->bytes) [0830] ErrorExitVmsStatus (SS$_BUGCHECK, ErrorSanityCheck, FI_LI); [0831] [0832] denptr->vlen = length; [0833] } [0834] [0835] /*****************************************************************************/ [0836] /* [0837] WATCH the contents of the dictionary. If |which| is NULL all entries are [0838] reported. If |which| is a single character string then it should be an entry [0839] type character and only matching entries are reported. If |which| is a string [0840] beginning and entry type character and followed by a key string then report [0841] only that entry. [0842] */ [0843] [0844] void DictWatch [0845] ( [0846] DICT_STRUCT *dicptr, [0847] uchar *type, [0848] uchar *which [0849] ) [0850] { [0851] char *cptr; [0852] DICT_ENTRY_STRUCT *denptr; [0853] REQUEST_STRUCT *rqptr; [0854] [0855] /*********/ [0856] /* begin */ [0857] /*********/ [0858] [0859] rqptr = dicptr->RequestPtr; [0860] [0861] if (which == NULL || (type == NULL && which[0] == '*')) [0862] { [0863] WatchThis (WATCHITM(rqptr), WATCH_INTERNAL, [0864] "DICTIONARY size:!UL count:!UL bytes:!UL", [0865] dicptr->size, dicptr->count, dicptr->bytes); [0866] DictWatchEntry (NULL); [0867] } [0868] [0869] if (which && which[1]) [0870] { [0871] if (type == NULL) type = DICT_TYPE_INTERNAL; [0872] denptr = DictLookup (rqptr->rqDictPtr, type, which, -1); [0873] if (denptr != NULL) DictWatchEntry (denptr); [0874] return; [0875] } [0876] [0877] for (denptr = dicptr->head; denptr != NULL; denptr = denptr->flink) [0878] if (which == NULL || type == NULL || type[0] == denptr->type[0]) [0879] DictWatchEntry (denptr); [0880] } [0881] [0882] /*****************************************************************************/ [0883] /* [0884] WATCH the specified entry. Entry pointer as NULL resets the counter. [0885] */ [0886] [0887] void DictWatchEntry (DICT_ENTRY_STRUCT *denptr) [0888] [0889] { [0890] static int count = 0; [0891] [0892] char *cptr, *sptr; [0893] DICT_STRUCT *dicptr; [0894] [0895] /*********/ [0896] /* begin */ [0897] /*********/ [0898] [0899] if (denptr == NULL) [0900] { [0901] count = 0; [0902] return; [0903] } [0904] [0905] dicptr = denptr->DictPtr; [0906] [0907] if (denptr == dicptr->table[denptr->hash]) [0908] cptr = "[]"; [0909] else [0910] cptr = ".."; [0911] [0912] if (denptr->type[0] == DICT_TYPE_OPAQUE || [0913] denptr->type[0] == DICT_TYPE_OPAQUE_KEY) [0914] WatchDataFormatted ("ENTRY !3ZL !&C!3ZL!&C !&C {!UL}!-!#&H={!UL}!-!#&H\n", [0915] ++count, cptr[0], denptr->hash, cptr[1], [0916] denptr->type[0], denptr->klen, denptr->key, [0917] denptr->vlen, denptr->value); [0918] else [0919] { [0920] if (denptr->vlen && denptr->value[denptr->vlen-1] == '\n') [0921] sptr = ""; [0922] else [0923] sptr = "\n"; [0924] WatchDataFormatted ("ENTRY !3ZL !&C!3ZL!&C !&C {!UL}!-!#AZ={!UL}!-!#AZ!AZ", [0925] ++count, cptr[0], denptr->hash, cptr[1], [0926] denptr->type[0], denptr->klen, denptr->key, [0927] denptr->vlen, denptr->value, sptr); [0928] } [0929] } [0930] [0931] /*****************************************************************************/ [0932] /* [0933] Exercise the dictionary functions. Provide some indicative timings. [0934] fname= should be a list of words such as found at: [0935] https://raw.githubusercontent.com/first20hours/google-10000-english/master/20k.txt [0936] [0937] Some results on an Alpha PWS500 OpenVMS V8.3. [0938] [0939] $! 8 hash, 1,000 additions results in a rate of 85k insertions/sec [0940] $ httpd /tests dictionary=size=8,count=10000,fname=20kwords.txt [0941] 8< snip 8< [0942] size:8 count:1000 max:0 min:0 [0943] total:1000 klen0:62 failures:62 duplicates:53 expands:32 deletions:0 duration:0.011718 85338.797/sec 0.000012/each [0944] table:[0]100 [1]99 [2]123 [3]119 [4]108 [5]109 [6]114 [7]103 [0945] [0946] $! 16 hash, 1,000 additions results in 102k insertions/sec [0947] $ httpd /tests dictionary=size=16,count=10000,fname=20kwords.txt [0948] 8< snip 8< [0949] size:16 count:1000 max:0 min:0 [0950] total:1000 klen0:57 failures:57 duplicates:52 expands:49 deletions:0 duration:0.009765 102406.555/sec 0.000010/each [0951] table:[0]57 [1]59 [2]61 [3]55 [4]48 [5]67 [6]51 [7]57 [8]56 [9]52 [10]44 [11]67 [12]51 [13]63 [14]39 [15]51 [0952] [0953] $! 32 hash, 100 additions, results in 114k insertions/sec [0954] $ httpd /tests dictionary=size=32,count=1000,fname=20kwords.txt [0955] 8< snip 8< [0956] size:32 count:1000 max:0 min:0 [0957] total:1000 klen0:58 failures:58 duplicates:46 expands:46 deletions:0 duration:0.008788 113785.062/sec 0.000009/each [0958] table:[0]28 [1]37 [2]25 [3]23 [4]30 [5]32 [6]31 [7]23 [8]30 [9]26 [10]28 [11]23 [12]25 [13]29 [14]27 [15]27 [16]28 [17]25 [18]27 [0959] [19]34 [20]30 [21]33 [22]19 [23]29 [24]25 [25]29 [26]27 [27]30 [28]22 [29]22 [30]35 [31]28 [0960] [0961] $! 64 hash, 100 additions, results in 114k insertions/sec [0962] $ httpd /tests dictionary=size=32,count=1000,fname=20kwords.txt [0963] 8< snip 8< [0964] size:64 count:1000 max:0 min:0 [0965] total:1000 klen0:60 failures:60 duplicates:51 expands:39 deletions:0 duration:0.008788 113785.062/sec 0.000009/each [0966] table:[0]14 [1]12 [2]13 [3]20 [4]17 [5]12 [6]14 [7]12 [8]13 [9]13 [10]16 [11]10 [12]11 [13]20 [14]13 [15]16 [16]14 [17]12 [18]10 [0967] [19]14 [20]16 [21]15 [22]16 [23]8 [24]15 [25]9 [26]15 [27]10 [28]11 [29]14 [30]9 [31]11 [32]10 [33]16 [34]9 [35]13 [36]10 [37]15 [0968] [38]12 [39]15 [40]5 [41]17 [42]14 [43]13 [44]21 [45]20 [46]18 [47]20 [48]17 [49]12 [50]16 [51]12 [52]10 [53]14 [54]12 [55]14 [0969] [56]12 [57]17 [58]15 [59]18 [60]13 [61]17 [62]16 [63]13 [0970] */ [0971] [0972] #if WATCH_MOD [0973] [0974] void DictTest (char *cliptr) [0975] [0976] { [0977] #define KEY_SIZE 31 [0978] #define KEY_VARY(ran) (ran & 0x1f) [0979] #define VALUE_SIZE 127 [0980] #define VALUE_VARY(ran) (ran & 0x7f) [0981] [0982] #include [0983] #include [0984] [0985] int count, deletions, duplicates, expands, failures, [0986] klen, klen0, max, min, size, status, total, vlen; [0987] int64 random, [0988] DeltaTime64, [0989] NowTime64, [0990] ThenTime64; [0991] uchar ch; [0992] uchar *bptr, *cptr, *fname, *sptr, *wptr, *zptr; [0993] uchar key [KEY_SIZE+1], [0994] value [VALUE_SIZE+1]; [0995] float fsecs; [0996] stat_t statbuf; [0997] FILE *fptr; [0998] DICT_STRUCT *dicptr; [0999] DICT_ENTRY_STRUCT *denptr; [1000] REQUEST_STRUCT *rqptr; [1001] [1002] /*********/ [1003] /* begin */ [1004] /*********/ [1005] [1006] for (cptr = cliptr; *cptr; cptr++) *cptr = to_lower(*cptr); [1007] [1008] size = 0; [1009] cptr = strstr (cliptr, "size="); [1010] if (cptr) size = atoi(cptr+5); [1011] if (size <= 0) size = DICT_DEFAULT_SIZE; [1012] [1013] count = 0; [1014] cptr = strstr (cliptr, "count="); [1015] if (cptr) count = atoi(cptr+6); [1016] if (count <= 0) count = size * 10; [1017] [1018] /* maximum number of entries before a deletion cycle */ [1019] max = 0; [1020] cptr = strstr (cliptr, "max="); [1021] if (cptr) max = atoi(cptr+4); [1022] [1023] /* minimum number of entries at end of delete cycle */ [1024] min = 0; [1025] cptr = strstr (cliptr, "min="); [1026] if (cptr) min = atoi(cptr+4); [1027] if (min < 0) min = 32; [1028] if (max < min) max = min * 2; [1029] [1030] cptr = strstr (cliptr, "fname="); [1031] if (cptr) fname = cptr+6; [1032] [1033] /* read in the file of strings */ [1034] if (stat (fname, &statbuf) < 0) exit (vaxc$errno); [1035] if ((fptr = fopen (fname, "r")) == NULL) exit (vaxc$errno); [1036] bptr = calloc (1, statbuf.st_size); [1037] if ((wptr = bptr) == NULL) exit (vaxc$errno); [1038] while (fgets (wptr, 256, fptr) != NULL) [1039] { [1040] while (*wptr && *wptr != '\n') wptr++; [1041] *wptr++ = ' '; [1042] *wptr = '\0'; [1043] } [1044] fclose (fptr); [1045] if (!*bptr) exit (SS$_ABORT); [1046] [1047] VmRequestInit (); [1048] rqptr = VmGetRequest (999999999); [1049] if (Watch.CliEnabled) [1050] { [1051] Watch.Category = Watch.Module = -1; [1052] WatchSetWatch (rqptr, WATCH_NEW_ITEM); [1053] } [1054] [1055] dicptr = DictCreate (rqptr, size); [1056] [1057] sys$gettim (&ThenTime64); [1058] random = ThenTime64 & (ThenTime64 << 40); [1059] [1060] deletions = duplicates = expands = failures = klen0 = 0; [1061] for (total = 0; total < count; total++) [1062] { [1063] random = random * 69069 + 1; [1064] /* ensure 5% (or so) are duplicate keys to exercise that aspect */ [1065] if (random % 21) [1066] { [1067] zptr = sptr = key; [1068] zptr += KEY_VARY(random) - 1; /* vary size */ [1069] while (sptr < zptr) [1070] { [1071] if (!*wptr) wptr = bptr; [1072] *sptr++ = *wptr++; [1073] } [1074] *sptr = '\0'; [1075] klen = sptr - key; [1076] } [1077] else [1078] duplicates++; [1079] [1080] /* generate random value */ [1081] random = random * 69069 + 1; [1082] zptr = sptr = value; [1083] zptr += VALUE_VARY(random) - 1; /* vary size */ [1084] while (sptr < zptr) [1085] { [1086] if (!*wptr) wptr = bptr; [1087] *sptr++ = *wptr++; [1088] } [1089] *sptr = '\0'; [1090] vlen = sptr - value; [1091] [1092] /* insert the key into the dictionary either of the two calls */ [1093] if (random % 2) [1094] denptr = DictInsert (dicptr, DICT_TYPE_INTERNAL, key, -1, value, -1); [1095] else [1096] denptr = DictInsert (dicptr, DICT_TYPE_INTERNAL, key, klen, value, vlen); [1097] [1098] /* a zero length key should fail */ [1099] if (denptr == NULL) failures++; [1100] if (klen == 0) klen0++; [1101] [1102] /* ensure another small percentage are expanded by 100% */ [1103] if ((random % 21) == 0) [1104] if (denptr != NULL) [1105] { [1106] DictExpandEntry (denptr, DICT_GET_BYTES(denptr) * 2); [1107] expands++; [1108] } [1109] [1110] /* if no deletion cycle */ [1111] if (!(max && min)) [1112] if (total < count) [1113] continue; [1114] else [1115] break; [1116] [1117] if (dicptr->count > max || total >= count) [1118] { [1119] DictIterate (dicptr, NULL); [1120] while (dicptr->count > min) [1121] { [1122] /* trim back down to min by choosing entries at random */ [1123] random = random * 69069 + 1; [1124] for (int cnt = (random & 0xf) + 1; cnt; cnt--) [1125] { [1126] while (denptr = DictIterate (dicptr, "*")); [1127] if (denptr) break; [1128] if ((denptr = DictIterate (dicptr, NULL)) == NULL) break; [1129] } [1130] if (denptr == NULL) break; [1131] if (dicptr->count % 2) [1132] DictRemove (dicptr, DICT_TYPE_INTERNAL, denptr->key, -1); [1133] else [1134] DictRemove (dicptr, DICT_TYPE_INTERNAL, [1135] denptr->key, denptr->klen); [1136] deletions++; [1137] } [1138] if (denptr == NULL) break; [1139] } [1140] if (denptr == NULL) break; [1141] } [1142] [1143] sys$gettim (&NowTime64); [1144] DeltaTime64 = ThenTime64 - NowTime64; [1145] fsecs = FloatDeltaTime (&DeltaTime64); [1146] [1147] Watch.LogFile = 1; [1148] strzcpy (Watch.LogFileName, WATCH_NAME_DEFAULT, sizeof(Watch.LogFileName)); [1149] Watch.Category = WATCH_INTERNAL; [1150] DictWatch (dicptr, NULL, NULL); [1151] [1152] fprintf (stdout, [1153] "size:%d count:%d max:%d min:%d\n\ [1154] total:%d klen0:%d failures:%d duplicates:%d expands:%d deletions:%d \ [1155] duration:%s %3.3f/sec %f/each\n", [1156] size, count, max, min, [1157] total, klen0, failures, duplicates, expands, deletions, [1158] DurationString (rqptr, &DeltaTime64), [1159] (float)total/fsecs, fsecs/(float)total); [1160] [1161] fputs ("table:", stdout); [1162] for (int idx = 0; idx < dicptr->size; idx++) [1163] { [1164] int cnt = 0; [1165] for (denptr = dicptr->table[idx]; denptr != NULL; denptr = denptr->clink) [1166] cnt++; [1167] fprintf (stdout, "[%d]%d ", idx, cnt); [1168] } [1169] fputs ("\n", stdout); [1170] [1171] DictDestroy (dicptr); [1172] } [1173] [1174] #endif /* WATCH_MOD */ [1175] [1176] /*****************************************************************************/ [1177]