Add casts to `bool`, or use `bit` type in local variables, where
neccessary to avoid MSVC compiler warnings C4242.
Note: There may remain places needing the above updates, where not
tested in CI, and missed in manual review.
Also:
- urldata: convert struct field `connect_only` to bitfield to match its
counterpart in another struct.
- rename curl-specific `bit` type to `curl_bit`.
Closes#20142
When a download size is known and rate limiting is in effect, adjust the
duration of each measurement step and its rate for maximum precision.
Since it is unpredictable how long the last bytes of a download will
take, download speed can be thrown off if the "last bytes" are a
significant amount of the total download. Make the "last bytes" small in
comparision to the rest and "stretch" the rate limit intervals to
accommodate the difference.
Fix ngtcp2 receive data acknowldgements to be based on a local window
size tracking. This allows window updates controlled by rate limits.
Fix ratelimit wait time calculation to accomodate negative tokens.
h3 rate limit, update timeers
Make download rate limits work correctly in ngtcp2. Fix multi handling
of rate limits to set a timer for when limits will update again.
Without running the transfer on limit updates, protocols like h2/h3 may
stall if the server does not send due to stream windows being too small.
scorecard: measure download speedlimits
When running scorecard with --limit-rate=n, show the reported download
speed averages plus percentage deviation from the limit.
Closes#20228
Most by moving functions around. Also delete unused ones.
Reducing their number from 83 to 33.
Remaining ones due to:
- circular dependencies.
- H3 code, that I did not attempt to update and likely the above applies.
- static declarations with attributes (`CURL_PRINTF`, `WARN_UNUSED_RESULT`).
- OS400 code.
Closes#20321
- asyn-thrdd.c: scope an include.
- apply more clang-format suggestions.
- tidy-up PP guard comments.
- delete empty line from the top of headers.
- add empty line after `curl_setup.h` include where missing.
- fix indent.
- CODE_STYLE.md: add `strcpy`.
Follow-up to 8636ad55df#20088
- lib1901.c: drop unnecessary line.
Follow-up to 436e67f65b#20076Closes#20070
- replace `sendf.h` with `curl_trc.h` where it was included just for it.
- drop unused `curl_trc.h` includes.
- easy: delete obsolete comment about `send.h` include reason.
Also:
- move out `curl_trc.h` include from `sendf.h` and include it directly
in users, where not done already. To flatten the include tree and
to less rely on indirect includes.
- stop including `sendf.h` from other headers, replace it with forward
declaration of `Curl_easy`, as done already elsewhere.
Verified with an all non-unity CI run.
Closes#20061
To make it available for all files. Drop includes from individual
sources. This header was already included from most sources and not
specific to any internal subsystem.
Also to ensure that two system symbol redefines on Windows (`read()` and
`write()`) get applied to all sources. Move them to `curl_setup.h`.
Closes#20056
- curl_range: replace `sendf.h` with direct header dependency
`curl_trc.h`.
- drop `curl/curl.h` includes from internal sourcees in favor of the
include made from `curl_setup.h`. Replace it with the latter where
it's the only include.
- include `curl_setup.h` before using macros, where missing.
- drop redundant `stdlib.h`, `string.h` includes, in favor of
`curl_setup_once.h` including them.
- drop redundant `limits.h` in favor of `curl_setup.h` including it.
- fake_addrinfo.h: fix typo in comment.
- curl_setup_once.h: drop `stdio.h` in favor of earlier include in
`curl_setup.h`.
- drop stray, unused, `stddef.h` includes.
- memdebug.h: add missing `stddef.h` include. (relying on accidental
includes via other headers before this patch.)
- stddef.h: document why it's included.
- strerr: drop `curl/mprintf.h` in favor of `curl/curl.h` including it
via `curl_setup.h`.
Closes#20027
Always use curlx_now() when calling Curl_pgrs_now(data). Tests with the
"manual" updates to now proved differ more then 100ms in parallel testing.
Add `curlx_nowp()` to set current time into a struct curltime.
Add `curlx_ptimediff_ms() and friends, passing pointers.
Update documentation.
Closes#19998
Use `data->progress.now` as the timestamp of proecssing a transfer.
Update it on significant events and refrain from calling `curlx_now()`
in many places.
The problem this addresses is
a) calling curlx_now() has costs, depending on platform. Calling it
every time results in 25% increase `./runtest` duration on macOS.
b) we used to pass a `struct curltime *` around to save on calls, but
when some method directly use `curx_now()` and some use the passed
pointer, the transfer experienes non-linear time. This results in
timeline checks to report events in the wrong order.
By keeping a timestamp in the easy handle and updating it there, no
longer invoking `curlx_now()` in the "lower" methods, the transfer
can observer a steady clock progression.
Add documentation in docs/internals/TIME-KEEPING.md
Reported-by: Viktor Szakats
Fixes#19935Closes#19961
Since we no longer traverse the transfers attached to a connection,
change the sparse bitset to just a `uint32_t` counter.
This makes multi_ev the single user of sparse bitsets for transfers
using a socket and allocation failures are handled there correctly.
Refs #19818Closes#19836
A fix for the tests that took the longest:
- test_05: make the server close the HTTP/1.1 connection when
simulating an error during a download. This eliminates waiting
for a keepalive timeout
- test_02: pause tests with slightly smaller documents, eliminate
special setup for HTTP/2. We test stream window handling now
elsewhere already
- cli_hx_download: run look in 500ms steps instead of 1sec, resuming
paused tranfers earlier.
Closes#19809
Before this patch curl used the C preprocessor to override standard
memory allocation symbols: malloc, calloc, strdup, realloc, free.
The goal of these is to replace them with curl's debug wrappers in
`CURLDEBUG` builds, another was to replace them with the wrappers
calling user-defined allocators in libcurl. This solution needed a bunch
of workarounds to avoid breaking external headers: it relied on include
order to do the overriding last. For "unity" builds it needed to reset
overrides before external includes. Also in test apps, which are always
built as single source files. It also needed the `(symbol)` trick
to avoid overrides in some places. This would still not fix cases where
the standard symbols were macros. It was also fragile and difficult
to figure out which was the actual function behind an alloc or free call
in a specific piece of code. This in turn caused bugs where the wrong
allocator was accidentally called.
To avoid these problems, this patch replaces this solution with
`curlx_`-prefixed allocator macros, and mapping them _once_ to either
the libcurl wrappers, the debug wrappers or the standard ones, matching
the rest of the code in libtests.
This concludes the long journey to avoid redefining standard functions
in the curl codebase.
Note: I did not update `packages/OS400/*.c` sources. They did not
`#include` `curl_setup.h`, `curl_memory.h` or `memdebug.h`, meaning
the overrides were never applied to them. This may or may not have been
correct. For now I suppressed the direct use of standard allocators
via a local `.checksrc`. Probably they (except for `curlcl.c`) should be
updated to include `curl_setup.h` and use the `curlx_` macros.
This patch changes mappings in two places:
- `lib/curl_threads.c` in libtests: Before this patch it mapped to
libcurl allocators. After, it maps to standard allocators, like
the rest of libtests code.
- `units`: before this patch it mapped to standard allocators. After, it
maps to libcurl allocators.
Also:
- drop all position-dependent `curl_memory.h` and `memdebug.h` includes,
and delete the now unnecessary headers.
- rename `Curl_tcsdup` macro to `curlx_tcsdup` and define like the other
allocators.
- map `curlx_strdup()` to `_strdup()` on Windows (was: `strdup()`).
To fix warnings silenced via `_CRT_NONSTDC_NO_DEPRECATE`.
- multibyte: map `curlx_convert_*()` to `_strdup()` on Windows
(was: `strdup()`).
- src: do not reuse the `strdup` name for the local replacement.
- lib509: call `_strdup()` on Windows (was: `strdup()`).
- test1132: delete test obsoleted by this patch.
- CHECKSRC.md: update text for `SNPRINTF`.
- checksrc: ban standard allocator symbols.
Follow-up to b12da22db1#18866
Follow-up to db98daab05#18844
Follow-up to 4deea9396b#18814
Follow-up to 9678ff5b1b#18776
Follow-up to 10bac43b87#18774
Follow-up to 20142f5d06#18634
Follow-up to bf7375ecc5#18503
Follow-up to 9863599d69#18502
Follow-up to 3bb5e58c10#17827Closes#19626
Change the send parameter from `const void *` to `const uint8_t *` and
adapt calling code. Several had already unsigned chars and were casting.
Closes#19729
Change `inputbuff` parameter from `const char *` to `const uint8_t *` to
reflect the binary nature of the input bytes. Half the code was casting
unsigned char to signed already in calling.
Closes#19722
Add protocol handler flag `PROTOPT_CONN_REUSE` to indicate that the
protocol allows reusing connections for other tranfers. Add that
to all handlers that support it.
Create connections with `conn->bits.close = FALSE` and remove all
the `connkeep()` calls in protocol handlers setup/connect implementations.
`PROTOPT_CONN_REUSE` assures that the default behaviour applies
at the end of a transfer without need to juggle the close bit.
`conn->bits.close` now serves as an additional indication that a
connection cannot be reused. Only protocol handles that allow
reuse need to set it to override the default behaviour.
Remove all `connclose()` and `connkeep()` calls from connection
filters. Filters should not modify connection flags. They are
supposed to run in eyeballing situations where a filter is just
one of many determining the outcome.
Fix http response header handling to only honour `Connection: close`
for HTTP/1.x versions.
Closes#19333
Description of how this works in `docs/internal/RATELIMITS.ms`.
Notable implementation changes:
- KEEP_SEND_PAUSE/KEEP_SEND_HOLD and KEEP_RECV_PAUSE/KEEP_RECV_HOLD
no longer exist. Pausing is down via blocked the new rlimits.
- KEEP_SEND_TIMED no longer exists. Pausing "100-continue" transfers
is done in the new `Curl_http_perform_pollset()` method.
- HTTP/2 rate limiting implemented via window updates. When
transfer initiaiting connection has a ratelimit, adjust the
initial window size
- HTTP/3 ngtcp2 rate limitin implemnented via ack updates
- HTTP/3 quiche does not seem to support this via its API
- the default progress-meter has been improved for accuracy
in "current speed" results.
pytest speed tests have been improved.
Closes#19384
The parsing of the HTTP/1.1 formatted request into the h2/h3 header
structures should detect CURLOPT_CUSTOMREQUEST methods and forward them
correctly.
Add test_01_20 to verify
Fixes#19543
Reported-by: Omdahake on github
Closes#19563
- badwords.pl: add `-a` option to check all lines in source code files.
Before this patch indented lines were skipped (to avoid Markdown code
fences.)
- GHA/checksrc: use `-a` when verifying the source code.
- GHA/checksrc: disable `So` and `But` rules for source code.
- GHA/checksrc: add docs/examples to the verified sources.
- badwords.txt: delete 4 duplicates.
- badwords.txt: group and sort contractions.
- badwords.txt: allow ` url = `, `DIR`, `<file name`.
Closes#19536
Add new functions in `curlx/warnless.h` for controlled type
conversions:
* curlx_uitouz, convert unsigned into to size_t (should always work)
* curlx_uztoso, convert size_t to curl_off_t, capping at CURL_OFF_T_MAX
* curlx_sztouz, convert ssize_t to size_t, return TRUE when ok
* curlx_sotouz_range, convert curl_off_t to size_t interval, capping
values to interval bounds
Remove some unnecesary casts, convert some internal recv functions
to the "return result, have size_t* arg" pattern.
Closes#19495
After this patch, the codebase no longer overrides system printf
functions. Instead it explicitly calls either the curl printf functions
`curl_m*printf()` or the system ones using their original names.
Also:
- drop unused `curl_printf.h` includes.
- checksrc: ban system printf functions, allow where necessary.
Follow-up to db98daab05#18844
Follow-up to 4deea9396b#18814Closes#18866
Improvements around HTTP Upgrade: and multiplex hanndling:
* add `Curl_conn_set_multiplex()` to set connection's multiplex
bit and trigger "connchanged" events
* call `Curl_conn_set_multiplex()` in filters' `CF_CTRL_CONN_INFO_UPDATE`
implementation where other connection properties are updated.
This prevents connection updates before the final filter chain
is chosen.
* rename enum `UPGR101_INIT` to `UPGR101_NONE`
* rename connection bit `asks_multiplex` to `upgrade_in_progress`
* trigger "connchanged" when `upgrade_in_progress` clears
* rename `WebSockets` to `WebSocket` as it is the common term
used in documentation
Closes#18227
Split out adding of individual request headers into a switch. Check
the connection http version only on fresh connections, use separate
methods.
Add TE: header directly without allocation. Add bit for indicating
Connection: header has been added and custom headers should not do
that again.
Closes#18444
`getsock()` calls operated on a global limit that could
not be configure beyond 16 sockets. This is no longer adequate
with the new happy eyeballing strategy.
Instead, do the following:
- make `struct easy_pollset` dynamic. Starting with
a minimal room for two sockets, the very common case,
allow it to grow on demand.
- replace all protocol handler getsock() calls with pollsets
and a CURLcode to return failures
- add CURLcode return for all connection filter `adjust_pollset()`
callbacks, since they too can now fail.
- use appropriately in multi.c and multi_ev.c
- fix unit2600 to trigger pollset growth
Closes#18164
Fix compile error when building with `--disable-verbose`.
Adjust pytest to skip when curl is not a debug build but needs
traces.
Follow-up to b453a447ceCloses#18053
- vtls: fix unused variable and symbols.
- ftp: fix unused variable.
- http: fix unused variables.
- smtp: fix unsued variable.
- wolfssl: fix unused variable with !proxy.
- libssh: fix unused argument.
- curl_trc: sync guards between declaration and definition.
- curl_trc: add missing guard for `Curl_trc_ssls` when !verbose.
- curl_trc: fix errors with !http + http3.
- curl_trc: fix missing function with !http + nghttp2.
- cf-h2-proxy: disable when !http + nghttp2, to avoid calling undeclared
functions.
- sha256: fix missing declaration in rare configs.
- md4: fix symbol conflict when building GnuTLS together with AWS-LC or
wolfSSL. By prioritizing the latter two. AWS-LC has no option
to disable the clashing symbol. wolfSSL does, but the most seamless is
to skip including GnuTLS's standalone `md4.h` to avoid the clash.
- build: fix errors with !http + nghttp2.
- build: catch !ssl + ssls-export combination in source. Convert
build-level errors to warnings.
- build: fix errors with !http + http3.
- build: fix building curl tool and unit1302 in rare combinations.
By always compiling base64 curlx functions.
- cmake: add `_CURL_SKIP_BUILD_CERTS` internal option.
To disable automatically building certs with the testdeps target.
To improve performance when testing builds.
(used locally to find the failing builds fixed in this PR.)
Closes#17962
- codespell: break logic out into its own runnable script. Allowing
to run it on local machines.
- codespell: install via `pip`, bump to latest version.
- codespell: show version number in CI log.
- codespell: drop no longer needed word exception: `msdos`.
- codespell: include all curl source tree, except `packages` and
`winbuild`. Drop an obsolete file exclusion.
- add new spellchecker job using the `typos` tool. It includes
the codespell dictionary and a couple more. Use linuxbrew to install
it. This takes 10 seconds, while installing via `cargo` from source
would take over a minute.
- codespell: introduce an inline ignore filter compatible with `cspell`
Make `typos` recognize it, too. Move single exceptions inline.
Fix new typos found. Also rename variables and words to keep
spellchecking exceptions at minumum. This involves touching some tests.
Also switch base64 strings to `%b64[]` to avoid false positives.
Ref: https://github.com/crate-ci/typos/blob/master/docs/reference.md
Ref: https://github.com/codespell-project/codespell?tab=readme-ov-file#inline-ignore
Ref: https://github.com/codespell-project/codespell/issues/1212#issuecomment-1721152455
Ref: https://cspell.org/docs/Configuration/document-settingsCloses#17905
Drop `strcasecompare` and `strncasecompare` in favor of libcurl API
calls `curl_strequal` and `curl_strnequal` respectively.
Also drop unnecessary `strcase.h` includes. Include `curl/curl.h`
instead where it wasn't included before.
Closes#17772
