90 Commits

Author SHA1 Message Date
Daniel Stenberg
6c31df453b mqtt: initial support for MQTTS
Closes #19418
2026-01-17 22:43:36 +01:00
Daniel Stenberg
dac98ccfa2 mqtt: better too-big-message-check
Reported-by: gudyuu on hackerone
Reviewed-by: Daniel Gustafsson
Bug: https://hackerone.com/reports/3508500
Closes #20281
2026-01-13 09:46:49 +01:00
Viktor Szakats
07926b5982 lib: reorder protocol functions to avoid forward declarations (misc)
For protocols: dict, file, gopher, tftp, http, mqtt, smb.

Move protocol hander table to the end of sources, rearrange static
functions is reverse dependency order as necessary.

Closes #20274
2026-01-13 01:31:27 +01:00
Daniel Stenberg
84e43eefc8 mqtt: return error when a too large packet is decoded
Closes #20148
2026-01-01 23:30:10 +01:00
Viktor Szakats
7032982896 tidy-up: miscellaneous
- asyn-thrdd.c: scope an include.
- apply more clang-format suggestions.
- tidy-up PP guard comments.
- delete empty line from the top of headers.
- add empty line after `curl_setup.h` include where missing.
- fix indent.
- CODE_STYLE.md: add `strcpy`.
  Follow-up to 8636ad55df #20088
- lib1901.c: drop unnecessary line.
  Follow-up to 436e67f65b #20076

Closes #20070
2025-12-26 22:06:09 +01:00
Viktor Szakats
e8415a8296 lib: drop, or replace sendf.h with curl_trc.h where possible
- replace `sendf.h` with `curl_trc.h` where it was included just for it.
- drop unused `curl_trc.h` includes.
- easy: delete obsolete comment about `send.h` include reason.

Also:
- move out `curl_trc.h` include from `sendf.h` and include it directly
  in users, where not done already. To flatten the include tree and
  to less rely on indirect includes.
- stop including `sendf.h` from other headers, replace it with forward
  declaration of `Curl_easy`, as done already elsewhere.

Verified with an all non-unity CI run.

Closes #20061
2025-12-21 12:39:25 +01:00
Viktor Szakats
884b5ea921 lib: include curlx/warnless.h from curl_setup.h
To make it available for all files. Drop includes from individual
sources. This header was already included from most sources and not
specific to any internal subsystem.

Also to ensure that two system symbol redefines on Windows (`read()` and
`write()`) get applied to all sources. Move them to `curl_setup.h`.

Closes #20056
2025-12-21 02:36:33 +01:00
Viktor Szakats
d11b8593a2 build: drop duplicate include curl/curl.h and others
- curl_range: replace `sendf.h` with direct header dependency
  `curl_trc.h`.
- drop `curl/curl.h` includes from internal sourcees in favor of the
  include made from `curl_setup.h`. Replace it with the latter where
  it's the only include.
- include `curl_setup.h` before using macros, where missing.
- drop redundant `stdlib.h`, `string.h` includes, in favor of
  `curl_setup_once.h` including them.
- drop redundant `limits.h` in favor of `curl_setup.h` including it.
- fake_addrinfo.h: fix typo in comment.
- curl_setup_once.h: drop `stdio.h` in favor of earlier include in
  `curl_setup.h`.
- drop stray, unused, `stddef.h` includes.
- memdebug.h: add missing `stddef.h` include. (relying on accidental
  includes via other headers before this patch.)
- stddef.h: document why it's included.
- strerr: drop `curl/mprintf.h` in favor of `curl/curl.h` including it
  via `curl_setup.h`.

Closes #20027
2025-12-19 10:58:11 +01:00
Viktor Szakats
fb50214604 build: drop unused multiif.h includes
Closes #20023
2025-12-19 01:45:05 +01:00
Stefan Eissing
b4be1f271e time-keeping: keep timestamp in multi, always update
Always use curlx_now() when calling Curl_pgrs_now(data). Tests with the
"manual" updates to now proved differ more then 100ms in parallel testing.

Add `curlx_nowp()` to set current time into a struct curltime.
Add `curlx_ptimediff_ms() and friends, passing pointers.

Update documentation.

Closes #19998
2025-12-18 22:10:06 +01:00
Stefan Eissing
2de22a00c7 lib: keep timestamp in easy handle
Use `data->progress.now` as the timestamp of proecssing a transfer.
Update it on significant events and refrain from calling `curlx_now()`
in many places.

The problem this addresses is
a) calling curlx_now() has costs, depending on platform. Calling it
   every time results in 25% increase `./runtest` duration on macOS.
b) we used to pass a `struct curltime *` around to save on calls, but
   when some method directly use `curx_now()` and some use the passed
   pointer, the transfer experienes non-linear time. This results in
   timeline checks to report events in the wrong order.

By keeping a timestamp in the easy handle and updating it there, no
longer invoking `curlx_now()` in the "lower" methods, the transfer
can observer a steady clock progression.

Add documentation in docs/internals/TIME-KEEPING.md

Reported-by: Viktor Szakats
Fixes #19935
Closes #19961
2025-12-16 08:48:44 +01:00
Viktor Szakats
c3b030b860 lib: fix formatting nits (part 3)
From `lib/h` to `lib/w`.

part 1: 47a1ab2ebe #19764
part 2: 86b346443b #19800

Closes #19811
2025-12-03 14:50:16 +01:00
Viktor Szakats
193cb00ce9 build: stop overriding standard memory allocation functions
Before this patch curl used the C preprocessor to override standard
memory allocation symbols: malloc, calloc, strdup, realloc, free.
The goal of these is to replace them with curl's debug wrappers in
`CURLDEBUG` builds, another was to replace them with the wrappers
calling user-defined allocators in libcurl. This solution needed a bunch
of workarounds to avoid breaking external headers: it relied on include
order to do the overriding last. For "unity" builds it needed to reset
overrides before external includes. Also in test apps, which are always
built as single source files. It also needed the `(symbol)` trick
to avoid overrides in some places. This would still not fix cases where
the standard symbols were macros. It was also fragile and difficult
to figure out which was the actual function behind an alloc or free call
in a specific piece of code. This in turn caused bugs where the wrong
allocator was accidentally called.

To avoid these problems, this patch replaces this solution with
`curlx_`-prefixed allocator macros, and mapping them _once_ to either
the libcurl wrappers, the debug wrappers or the standard ones, matching
the rest of the code in libtests.

This concludes the long journey to avoid redefining standard functions
in the curl codebase.

Note: I did not update `packages/OS400/*.c` sources. They did not
`#include` `curl_setup.h`, `curl_memory.h` or `memdebug.h`, meaning
the overrides were never applied to them. This may or may not have been
correct. For now I suppressed the direct use of standard allocators
via a local `.checksrc`. Probably they (except for `curlcl.c`) should be
updated to include `curl_setup.h` and use the `curlx_` macros.

This patch changes mappings in two places:
- `lib/curl_threads.c` in libtests: Before this patch it mapped to
  libcurl allocators. After, it maps to standard allocators, like
  the rest of libtests code.
- `units`: before this patch it mapped to standard allocators. After, it
  maps to libcurl allocators.

Also:
- drop all position-dependent `curl_memory.h` and `memdebug.h` includes,
  and delete the now unnecessary headers.
- rename `Curl_tcsdup` macro to `curlx_tcsdup` and define like the other
  allocators.
- map `curlx_strdup()` to `_strdup()` on Windows (was: `strdup()`).
  To fix warnings silenced via `_CRT_NONSTDC_NO_DEPRECATE`.
- multibyte: map `curlx_convert_*()` to `_strdup()` on Windows
  (was: `strdup()`).
- src: do not reuse the `strdup` name for the local replacement.
- lib509: call `_strdup()` on Windows (was: `strdup()`).
- test1132: delete test obsoleted by this patch.
- CHECKSRC.md: update text for `SNPRINTF`.
- checksrc: ban standard allocator symbols.

Follow-up to b12da22db1 #18866
Follow-up to db98daab05 #18844
Follow-up to 4deea9396b #18814
Follow-up to 9678ff5b1b #18776
Follow-up to 10bac43b87 #18774
Follow-up to 20142f5d06 #18634
Follow-up to bf7375ecc5 #18503
Follow-up to 9863599d69 #18502
Follow-up to 3bb5e58c10 #17827

Closes #19626
2025-11-28 10:44:26 +01:00
Stefan Eissing
b812be567a mqtt: eliminate size_t cast
Use new curlx_sotouz_fits() instead.

Closes #19622
2025-11-20 17:39:39 +01:00
Stefan Eissing
78a610cb83 lib: rename curlx_timediff to curlx_timeleft_ms
Rename `Curl_timeleft()` to `Curl_timeleft_ms()` to make the units in
the returned `timediff_t` clear. (We used to always have ms there, but
with QUIC started to sometimes calc ns as well).

Rename some assigned vars without `_ms` suffix for clarity as well.

Closes #19486
2025-11-13 13:12:58 +01:00
x2018
87149c8383 mqtt: properly handle the message which exceeds maxsize
We should goto fail as topic is allocated.

Follow-up to 92fd791

Closes #19417
2025-11-10 09:07:27 +01:00
Daniel Stenberg
92fd791f31 mqtt: reject overly big messages
Reported-by: Jiyong Yang
Closes #19415
2025-11-09 11:40:28 +01:00
Viktor Szakats
a13d811044 lib: delete unused header includes
`escape.h`, `getinfo.h`, `strdup.h`.

Tested OK with full non-unity CI run.

Closes #19231
2025-10-25 00:19:00 +02:00
Viktor Szakats
b12da22db1 lib: stop overriding system printf symbols
After this patch, the codebase no longer overrides system printf
functions. Instead it explicitly calls either the curl printf functions
`curl_m*printf()` or the system ones using their original names.

Also:
- drop unused `curl_printf.h` includes.
- checksrc: ban system printf functions, allow where necessary.

Follow-up to db98daab05 #18844
Follow-up to 4deea9396b #18814

Closes #18866
2025-10-06 20:57:59 +02:00
Viktor Szakats
92f215fea1 build: address some -Weverything warnings, update picky warnings
`-Weverything` is not enabled by curl, and not recommended by LLVM,
because it may enable experimental options, and will result in new
fallouts after toolchain upgrades. This patch aims to fix/silence as much
as possible as found with llvm/clang 21.1.0. It also permanently enables
warnings that were fixed in source and deemed manageable in the future.
`-Wformat` warnings are addressed separately via #18343.

Fix/silence warnings in the source:
- typecheck-gcc.h: fix `-Wreserved-identifier`.
- lib: silence `-Wcast-function-type-strict`.
  For llvm 16+ or Apple clang 16+.
- asyn-ares: limit `HAPPY_EYEBALLS_DNS_TIMEOUT` to old c-ares versions.
- curl_trc: fix `-Wc++-hidden-decl`.
- doh: fix `-Wc++-keyword`.
- ftp: fix `-Wreserved-identifier`.
- ldap: fix `-Wreserved-identifier`.
- mqtt: comment unused macro to avoid warning.
- multi_ev: drop unused macros to avoid warnings.
- setopt: fix useless `break;` after `return;`.
- gtls, mbedtls, rustls: silence `-Wconditional-uninitialized`.
- socks_sspi, schannel, x509asn1: fix `-Wimplicit-int-enum-cast`.
- x509asn1: fix `-Wc++-keyword`.
- openssl: scope `OSSL_UI_METHOD_CAST` to avoid unused macro warning.
- libssh2, wolfssl: drop unused macros.
- curl_ngtcp2, curl_quiche, httpsrr, urlapi: drop/limit unused macros.
- tool_getparam: fix useless `break;` after `return;` or `break;`.
  Not normally enabled because it doesn't work with unity.
  https://github.com/llvm/llvm-project/issues/71046
- tool_operate: fix `-Wc++-keyword`.
- curlinfo: fix a `-Wunsafe-buffer-usage`.
- tests: silence `-Wformat-non-iso`.
- lib557: fix `-Wreserved-identifier`.
- lib1565: silence `-Wconditional-uninitialized`.

Enable the above clang warnings permanently in picky mode:
- `-Wc++-hidden-decl`
- `-Wc++-keyword` (except for Windows, where it collides with `wchar_t`)
- `-Wcast-function-type-strict`
- `-Wcast-function-type`
- `-Wconditional-uninitialized`
- `-Wformat-non-iso` (except for clang-cl)
- `-Wreserved-identifier`
- `-Wtentative-definition-compat`

Silence problematic `-Weverything` warnings globally (in picky mode):
- `-Wused-but-marked-unused` (88000+ hits) and
  `-Wdisabled-macro-expansion` (2600+ hits).
  Triggered by `typecheck-gcc.h` when building with clang 14+.
  Maybe there exists a way to fix within that header?
  Ref: https://discourse.llvm.org/t/removing-wused-but-marked-unused/55310
- `-Wunsafe-buffer-usage`. clang 16+. 7000+ hits.
  May be useful in theory, but such high volume of hits makes it
  impractical to review and possibly address. Meant for C++.
  Ref: https://clang.llvm.org/docs/SafeBuffers.html
  Ref: https://stackoverflow.com/questions/77017567/how-to-fix-code-to-avoid-warning-wunsafe-buffer-usage
  Ref: https://discourse.llvm.org/t/rfc-c-buffer-hardening/65734
  Ref: https://github.com/llvm/llvm-project/pull/111624
- `-Wimplicit-void-ptr-cast`. clang 21+. 1700+ hits.
  C++ warning, deemed pure noise.
  Ref: https://github.com/curl/curl/issues/18470#issuecomment-3253506266
- `-Wswitch-default` (180+ hits), `-Wswitch-enum` (190+ hits),
  `-Wcovered-switch-default` (20+ hits).
  Next to impossible to fix cleanly, esp. when the covered `case`
  branches depend on compile-time options.
- `-Wdocumentation-unknown-command` (8+ hits).
  Triggered in a few sources. Seems arbitrary and bogus.
- `-Wpadded` (550+ hits).
- `-Wc++-keyword` on Windows, where it collides with `wchar_t`.
  (100+ hits)
  Ref: https://github.com/llvm/llvm-project/issues/155988
- `-Wreserved-macro-identifier`. clang 13+. 5+ hits.
  Sometimes it's necessary to set external macros that use
  the reserved namespace. E.g. `_CRT_NONSTDC_NO_DEPRECATE`,
  `__ENVIRONMENT_MAC_OS_X_VERSION_MIN_REQUIRED__`, `__NO_NET_API`,
  possibly `_REENTRANT`, and more.
  It's not worth trying to silence them individually.
- `-Wnonportable-system-include-path` with `clang-cl`.
  It'd be broken by doing what the warning suggests.
- `-Wformat-non-iso` for clang-cl.

CMake `PICKY_COMPILER=ON` (the default) or `./configure`
`--enable-warnings` (not the default) is required to enable these
silencing rules.

Also:
- autotools, cmake: fix Apple clang and mainline llvm version translations.
  Ref: https://en.wikipedia.org/wiki/Xcode#Toolchain_versions
- autotools, cmake: enable `-Warray-compare` for clang 20+.
  Follow-up to 4b7accda5a #17196
- cmake: fix to enable `-Wmissing-variable-declarations` at an earlier
  clang version.
- cmake: update internal logic to handle warning options with `+` in
  them.
- cmake: fix internal logic to match the whole option when looking
  into `CMAKE_C_FLAGS` for custom-disabled warnings.

Follow-up to b85cb8cb4e #18485

Closes #18477
2025-09-20 10:16:15 +02:00
Stefan Eissing
5b80b4c012 lib: replace getsock() logic with pollsets
`getsock()` calls operated on a global limit that could
not be configure beyond 16 sockets. This is no longer adequate
with the new happy eyeballing strategy.

Instead, do the following:
- make `struct easy_pollset` dynamic. Starting with
  a minimal room for two sockets, the very common case,
  allow it to grow on demand.
- replace all protocol handler getsock() calls with pollsets
  and a CURLcode to return failures
- add CURLcode return for all connection filter `adjust_pollset()`
  callbacks, since they too can now fail.
- use appropriately in multi.c and multi_ev.c
- fix unit2600 to trigger pollset growth

Closes #18164
2025-08-04 23:43:13 +02:00
Viktor Szakats
8eab2b7086 tidy-up: whitespace
Cherry-picked from #17877
Cherry-picked from #17876

Closes #17896
2025-07-11 13:32:54 +02:00
Stefan Eissing
20c90ba298 lib: unify recv/send function signatures
cfilter/conn: change send/recv function signatures. Unify the
calling/return conventions in our send/receive handling.

Curl_conn_recv(), adjust pnread type

Parameter `pnread` was a `ssize_t *`, but `size_t *` is better since the
function returns any error in its `CURLcode` return value.

Closes #17546
2025-06-11 11:21:10 +02:00
Daniel Stenberg
255aac56f9 curlx: move into to curlx/
Move curlx_ functions into its own subdir.

The idea is to use the curlx_ prefix proper on these functions, and use
these same function names both in tool, lib and test suite source code.
Stop the previous special #define setup for curlx_ names.

The printf defines are now done for the library alone. Tests no longer
use the printf defines. The tool code sets its own defines. The printf
functions are not curlx, they are publicly available.

The strcase defines are not curlx_ functions and should not be used by
tool or server code.

dynbuf, warnless, base64, strparse, timeval, timediff are now proper
curlx functions.

When libcurl is built statically, the functions from the library can be
used as-is. The key is then that the functions must work as-is, without
having to be recompiled for use in tool/tests. This avoids symbol
collisions - when libcurl is built statically, we use those functions
directly when building the tool/tests. When libcurl is shared, we
build/link them separately for the tool/tests.

Assisted-by: Jay Satiro

Closes #17253
2025-05-07 11:01:15 +02:00
Stefan Eissing
47b2300192 mqtt: use conn/easy meta hash
Remove mqtt structs from the unions at connectdata and
easy handle requests. Use meta hash at easy/connnection.

Make mqtt structs private to mqtt.c

Closes #17221
2025-04-29 14:25:25 +02:00
Christian Schmitz
8ad0243e1f mqtt: send ping at upkeep interval
Closes #16975
2025-04-16 09:36:19 +02:00
Daniel Stenberg
9a7baa34cc mqtt: cleanups
- make a variable const
- use calloc instead of malloc + memset
- fix comment language
- improve error messages

Closes #16826
2025-03-25 09:18:08 +01:00
Daniel Stenberg
b2078c0e1c mqtt: convert sendleftovers to dynbuf
Avoid frequent strdups/free calls, including the double-free risk.

Reported-by: Ronald Crane
Closes #16823
2025-03-25 08:22:05 +01:00
Viktor Szakats
f4e23950c7 build: enable -Wcast-qual, fix or silence compiler warnings
The issues found fell into these categories, with the applied fixes:

- const was accidentally stripped.
  Adjust code to not cast or cast with const.

- const/volatile missing from arguments, local variables.
  Constify arguments or variables, adjust/delete casts. Small code
  changes in a few places.

- const must be stripped because an API dependency requires it.
  Strip `const` with `CURL_UNCONST()` macro to silence the warning out
  of our control. These happen at API boundaries. Sometimes they depend
  on dependency version, which this patch handles as necessary. Also
  enable const support for the zlib API, using `ZLIB_CONST`. Supported
  by zlib 1.2.5.2 and newer.

- const must be stripped because a curl API requires it.
  Strip `const` with `CURL_UNCONST()` macro to silence the warning out
  of our immediate control. For example we promise to send a non-const
  argument to a callback, though the data is const internally.

- other cases where we may avoid const stripping by code changes.
  Also silenced with `CURL_UNCONST()`.

- there are 3 places where `CURL_UNCONST()` is cast again to const.
  To silence this type of warning:
  ```
  lib/vquic/curl_osslq.c:1015:29: error: to be safe all intermediate
    pointers in cast from 'unsigned char **' to 'const unsigned char **'
    must be 'const' qualified [-Werror=cast-qual]
  lib/cf-socket.c:734:32: error: to be safe all intermediate pointers in
    cast from 'char **' to 'const char **' must be 'const' qualified
    [-Werror=cast-qual]
  ```
  There may be a better solution, but I couldn't find it.

These cases are handled in separate subcommits, but without further
markup.

If you see a `-Wcast-qual` warning in curl, we appreciate your report
about it.

Closes #16142
2025-03-10 22:30:15 +01:00
Viktor Szakats
784a8ec2c1 tidy-up: delete, comment or scope C macros reported unused
To reduce the number `-Wunused-macro` compiler warnings:
- delete unused macros.
- comment out unused macro that are part of a set.
- move macros into the scope they are used.

This may be useful to enable by default, but there are tricky cases that
I didn't manage to fix and paused the effort. E.g. internal features
checks in `openssl.c`. There is more, once those are fixed.

Closes #16279
2025-02-14 10:37:14 +01:00
Stefan Eissing
1213c31272 lib: redirect handling by protocol handler
Adds a `follow()` callback to protocol handlers, so they may decide how
to act on a `newurl` after a request has been done. This is optional.

This moves the HTTP code for handling redirects from multi.c to http.c
where it should be. If we ever add a protocol with its own logic, it
would install its own follow function.

Closes #16075
2025-01-24 11:00:34 +01:00
Daniel Stenberg
fbf5d507ce lib/src: white space edits to comply better with code style
... as checksrc now finds and complains about these.

Closes #14921
2024-09-19 14:59:12 +02:00
Stefan Eissing
911c3166b6 lib: add eos flag to send methods
Adds a `bool eos` flag to send methods to indicate that the data
is the last chunk the invovled transfer wants to send to the server.

This will help protocol filters like HTTP/2 and 3 to forward the
stream's EOF flag and also allow to EAGAIN such calls when buffers
are not yet fully flushed.

Closes #14220
2024-08-03 19:53:54 +02:00
Daniel Stenberg
25321de30e Revert "lib: send eos flag"
This reverts commit be93299f10.
2024-07-19 01:38:05 +02:00
Stefan Eissing
be93299f10 lib: send eos flag
Adds a `bool eos` flag to send methods to indicate that the data is the
last chunk the invovled transfer wants to send to the server.

This will help protocol filters like HTTP/2 and 3 to forward the
stream's EOF flag and also allow to EAGAIN such calls when buffers are
not yet fully flushed.

Closes #14220
2024-07-18 23:27:35 +02:00
Daniel Stenberg
c074ba64a8 code: language cleanup in comments
Based on the standards and guidelines we use for our documentation.

 - expand contractions (they're => they are etc)
 - host name = > hostname
 - file name => filename
 - user name = username
 - man page => manpage
 - run-time => runtime
 - set-up => setup
 - back-end => backend
 - a HTTP => an HTTP
 - Two spaces after a period => one space after period

Closes #14073
2024-07-01 22:58:55 +02:00
Viktor Szakats
d68a121266 cmake: improve wolfSSL detection
- support detecting wolfSSL via pkg-config (like autotools.)

- detect wolfSSL version.

- detect `HAVE_WOLFSSL_DES_ECB_ENCRYPT`.
  (needs e.g. `--enable-curl` when building wolfSSL)

- detect `HAVE_WOLFSSL_FULL_BIO` and enable HTTPS-proxy feature.
  (needs e.g. `--enable-opensslall` when building wolfSSL)

- fix to show `HTTPS-proxy` in cmake feature list.
  Ref: 55807e6c05 #9962

- fix to show `NTLM` in cmake feature list.

- fix to show `smb` and `smbs` in cmake protocol list.

- add wolfSSL CMake job to GHA (for macOS).

- fix mqtt and wolfSSL symbol clash.
  ```
  ./curl/lib/mqtt.c: In function 'mqtt_doing':
  ./curl/lib/mqtt.c:746:17: error: declaration of 'byte' shadows a global declaration [-Werror=shadow]
    746 |   unsigned char byte;
        |                 ^~~~
  /opt/homebrew/Cellar/wolfssl/5.7.0_1/include/wolfssl/wolfcrypt/types.h:85:36: note: shadowed declaration is here
     85 |             typedef unsigned char  byte;
        |                                    ^~~~
  ```

- format `FindWolfSSL.cmake` closer to neighbours.

Closes #14064
2024-07-01 17:49:14 +02:00
Viktor Szakats
72abf7c13a lib: tidy up types and casts
Cherry-picked from #13489
Closes #13862
2024-06-05 14:02:39 +02:00
Viktor Szakats
59dc9f7e69 build: untangle CURLDEBUG and DEBUGBUILD macros
`CURLDEBUG` is meant to enable memory tracking, but in a bunch of cases,
it was protecting debug features that were supposed to be guarded with
`DEBUGBUILD`.

Replace these uses with `DEBUGBUILD`.

This leaves `CURLDEBUG` uses solely for its intended  purpose: to enable
the memory tracking debug feature.

Also:
- autotools: rely on `DEBUGBUILD` to enable `checksrc`.
  Instead of `CURLDEBUG`, which worked in most cases because debug
  builds enable `CURLDEBUG` by default, but it's not accurate.
- include `lib/easyif.h` instead of keeping a copy of a declaration.
- add CI test jobs for the build issues discovered.

Ref: https://github.com/curl/curl/pull/13694#issuecomment-2120311894
Closes #13718
2024-05-28 08:12:00 +02:00
Daniel Stenberg
c294f9cb56 lib: make protocol handlers store scheme name lowercase
- saves a lowercase operation when the "[scheme]_proxy" name is
  generated
- appears less "shouting"
- update test 970, 972, 1438 and 1536

Closes #13553
2024-05-08 09:39:30 +02:00
Daniel Stenberg
123d3ef5db mqtt: when Curl_xfer_recv returns error, don't use nread
A returned error code makes other return value unreliable, and in this
case potentially uninitialized. On error, do not read other return
values like the nread counter.

Spotted by CodeSonar

Closes #13418
2024-04-19 13:30:09 +02:00
Stefan Eissing
8dd81bd5db lib: add Curl_xfer_write_resp_hd
Add method in protocol handlers to allow writing of a single,
0-terminated header line. Avoids parsing and copying these lines.

Closes #13165
2024-04-11 09:29:21 +02:00
Stefan Eissing
3755153571 lib: Curl_read/Curl_write clarifications
- replace `Curl_read()`, `Curl_write()` and `Curl_nwrite()` to
  clarify when and at what level they operate
- send/recv of transfer related data is now done via
  `Curl_xfer_send()/Curl_xfer_recv()` which no longer has
  socket/socketindex as parameter. It decides on the transfer
  setup of `conn->sockfd` and `conn->writesockfd` on which
  connection filter chain to operate.
- send/recv on a specific connection filter chain is done via
  `Curl_conn_send()/Curl_conn_recv()` which get the socket index
  as parameter.
- rename `Curl_setup_transfer()` to `Curl_xfer_setup()` for
  naming consistency
- clarify that the special CURLE_AGAIN hangling to return
  `CURLE_OK` with length 0 only applies to `Curl_xfer_send()`
  and CURLE_AGAIN is returned by all other send() variants.
- fix a bug in websocket `curl_ws_recv()` that mixed up data
  when it arrived in more than a single chunk

The method for sending not just raw bytes, but bytes that are either
"headers" or "body". The send abstraction stack, to to bottom, now is:

* `Curl_req_send()`: has parameter to indicate amount of header bytes,
  buffers all data.
* `Curl_xfer_send()`: knows on which socket index to send, returns
  amount of bytes sent.
* `Curl_conn_send()`: called with socket index, returns amount of bytes
  sent.

In addition there is `Curl_req_flush()` for writing out all buffered
bytes.

`Curl_req_send()` is active for requests without body,
`Curl_buffer_send()` still being used for others. This is because the
special quirks need to be addressed in future parts:

* `expect-100` handling
* `Curl_fillreadbuffer()` needs to add directly to the new
  `data->req.sendbuf`
* special body handlings, like `chunked` encodings and line end
  conversions will be moved into something like a Client Reader.

In functions of the pattern `CURLcode xxx_send(..., ssize_t *written)`,
replace the `ssize_t` with a `size_t`. It makes no sense to allow for negative
values as the returned `CURLcode` already specifies error conditions. This
allows easier handling of lengths without casting.

Closes #12964
2024-02-27 14:13:56 +01:00
Stefan Eissing
5929822114 lib: send rework
Curl_read/Curl_write clarifications

- replace `Curl_read()`, `Curl_write()` and `Curl_nwrite()` to 1clarify
  when and at what level they operate

- send/recv of transfer related data is now done via
  `Curl_xfer_send()/Curl_xfer_recv()` which no longer has
  socket/socketindex as parameter. It decides on the transfer setup of
  `conn->sockfd` and `conn->writesockfd` on which connection filter
  chain to operate.

- send/recv on a specific connection filter chain is done via
  `Curl_conn_send()/Curl_conn_recv()` which get the socket index as
  parameter.

- rename `Curl_setup_transfer()` to `Curl_xfer_setup()` for naming
  consistency

- clarify that the special CURLE_AGAIN handling to return `CURLE_OK`
  with length 0 only applies to `Curl_xfer_send()` and CURLE_AGAIN is
  returned by all other send() variants.

SingleRequest reshuffling

- move functions into request.[ch]
- differentiate between reset and free
- add Curl_req_done() to perform last actions
- add a send `bufq` to SingleRequest for future use in keeping upload data

Closes #12963
2024-02-27 08:58:10 +01:00
Stefan Eissing
bdae275958 mqtt, remove remaining use of data->state.buffer
Closes #12799
2024-01-27 12:42:19 +01:00
Stefan Eissing
73c4aaa2c0 mqtt: use stack scratch buffer for recv+publish
- instead of data->state.buffer

Closes #12792
2024-01-25 22:32:18 +01:00
Stefan Eissing
d7b6ce64ce lib: replace readwrite with write_resp
This clarifies the handling of server responses by folding the code for
the complicated protocols into their protocol handlers. This concerns
mainly HTTP and its bastard sibling RTSP.

The terms "read" and "write" are often used without clear context if
they refer to the connect or the client/application side of a
transfer. This PR uses "read/write" for operations on the client side
and "send/receive" for the connection, e.g. server side. If this is
considered useful, we can revisit renaming of further methods in another
PR.

Curl's protocol handler `readwrite()` method been changed:

```diff
-  CURLcode (*readwrite)(struct Curl_easy *data, struct connectdata *conn,
-                        const char *buf, size_t blen,
-                        size_t *pconsumed, bool *readmore);
+  CURLcode (*write_resp)(struct Curl_easy *data, const char *buf, size_t blen,
+                         bool is_eos, bool *done);
```

The name was changed to clarify that this writes reponse data to the
client side. The parameter changes are:

* `conn` removed as it always operates on `data->conn`
* `pconsumed` removed as the method needs to handle all data on success
* `readmore` removed as no longer necessary
* `is_eos` as indicator that this is the last call for the transfer
  response (end-of-stream).
* `done` TRUE on return iff the transfer response is to be treated as
  finished

This change affects many files only because of updated comments in
handlers that provide no implementation. The real change is that the
HTTP protocol handlers now provide an implementation.

The HTTP protocol handlers `write_resp()` implementation will get passed
**all** raw data of a server response for the transfer. The HTTP/1.x
formatted status and headers, as well as the undecoded response
body. `Curl_http_write_resp_hds()` is used internally to parse the
response headers and pass them on. This method is public as the RTSP
protocol handler also uses it.

HTTP/1.1 "chunked" transport encoding is now part of the general
*content encoding* writer stack, just like other encodings. A new flag
`CLIENTWRITE_EOS` was added for the last client write. This allows
writers to verify that they are in a valid end state. The chunked
decoder will check if it indeed has seen the last chunk.

The general response handling in `transfer.c:466` happens in function
`readwrite_data()`. This mainly operates now like:

```
static CURLcode readwrite_data(data, ...)
{
  do {
    Curl_xfer_recv_resp(data, buf)
    ...
    Curl_xfer_write_resp(data, buf)
    ...
  } while(interested);
  ...
}
```

All the response data handling is implemented in
`Curl_xfer_write_resp()`. It calls the protocol handler's `write_resp()`
implementation if available, or does the default behaviour.

All raw response data needs to pass through this function. Which also
means that anyone in possession of such data may call
`Curl_xfer_write_resp()`.

Closes #12480
2024-01-13 17:23:42 +01:00
Daniel Stenberg
cfe7902111 lib: add debug log outputs for CURLE_BAD_FUNCTION_ARGUMENT
Closes #12658
2024-01-08 22:48:24 +01:00
Viktor Szakats
3829759bd0 build: enable missing OpenSSF-recommended warnings, with fixes
https://best.openssf.org/Compiler-Hardening-Guides/Compiler-Options-Hardening-Guide-for-C-and-C++.html
as of 2023-11-29 [1].

Enable new recommended warnings (except `-Wsign-conversion`):

- enable `-Wformat=2` for clang (in both cmake and autotools).
- add `CURL_PRINTF()` internal attribute and mark functions accepting
  printf arguments with it. This is a copy of existing
  `CURL_TEMP_PRINTF()` but using `__printf__` to make it compatible
  with redefinting the `printf` symbol:
  https://gcc.gnu.org/onlinedocs/gcc-3.0.4/gcc_5.html#SEC94
- fix `CURL_PRINTF()` and existing `CURL_TEMP_PRINTF()` for
  mingw-w64 and enable it on this platform.
- enable `-Wimplicit-fallthrough`.
- enable `-Wtrampolines`.
- add `-Wsign-conversion` commented with a FIXME.
- cmake: enable `-pedantic-errors` the way we do it with autotools.
  Follow-up to d5c0351055 #2747
- lib/curl_trc.h: use `CURL_FORMAT()`, this also fixes it to enable format
  checks. Previously it was always disabled due to the internal `printf`
  macro.

Fix them:

- fix bug where an `set_ipv6_v6only()` call was missed in builds with
  `--disable-verbose` / `CURL_DISABLE_VERBOSE_STRINGS=ON`.
- add internal `FALLTHROUGH()` macro.
- replace obsolete fall-through comments with `FALLTHROUGH()`.
- fix fallthrough markups: Delete redundant ones (showing up as
  warnings in most cases). Add missing ones. Fix indentation.
- silence `-Wformat-nonliteral` warnings with llvm/clang.
- fix one `-Wformat-nonliteral` warning.
- fix new `-Wformat` and `-Wformat-security` warnings.
- fix `CURL_FORMAT_SOCKET_T` value for mingw-w64. Also move its
  definition to `lib/curl_setup.h` allowing use in `tests/server`.
- lib: fix two wrongly passed string arguments in log outputs.
  Co-authored-by: Jay Satiro
- fix new `-Wformat` warnings on mingw-w64.

[1] 56c0fde389/docs/Compiler-Hardening-Guides/Compiler-Options-Hardening-Guide-for-C-and-C%2B%2B.md

Closes #12489
2023-12-16 13:12:37 +00:00
Viktor Szakats
84338c4de2 build: add more picky warnings and fix them
Enable more picky compiler warnings. I've found these options in the
nghttp3 project when implementing the CMake quick picky warning
functionality for it [1].

`-Wunused-macros` was too noisy to keep around, but fixed a few issues
it revealed while testing.

- autotools: reflect the more precisely-versioned clang warnings.
  Follow-up to 033f8e2a08 #12324
- autotools: sync between clang and gcc the way we set `no-multichar`.
- autotools: avoid setting `-Wstrict-aliasing=3` twice.
- autotools: disable `-Wmissing-noreturn` for MSYS gcc targets [2].
  It triggers in libtool-generated stub code.

- lib/timeval: delete a redundant `!MSDOS` guard from a `WIN32` branch.

- lib/curl_setup.h: delete duplicate declaration for `fileno`.
  Added in initial commit ae1912cb0d
  (1999-12-29). This suggests this may not be needed anymore, but if
  it does, we may restore this for those specific (non-Windows) systems.
- lib: delete unused macro `FTP_BUFFER_ALLOCSIZE` since
  c1d6fe2aaa.
- lib: delete unused macro `isxdigit_ascii` since
  f65f750742.
- lib/mqtt: delete unused macro `MQTT_HEADER_LEN`.
- lib/multi: delete unused macro `SH_READ`/`SH_WRITE`.
- lib/hostip: add `noreturn` function attribute via new `CURL_NORETURN`
  macro.
- lib/mprintf: delete duplicate declaration for `Curl_dyn_vprintf`.
- lib/rand: fix `-Wunreachable-code` and related fallouts [3].
- lib/setopt: fix `-Wunreachable-code-break`.
- lib/system_win32 and lib/timeval: fix double declarations for
  `Curl_freq` and `Curl_isVistaOrGreater` in CMake UNITY mode [4].
- lib/warnless: fix double declarations in CMake UNITY mode [5].
  This was due to force-disabling the header guard of `warnless.h` to
  to reapply it to source code coming after `warnless.c` in UNITY
  builds. This reapplied declarations too, causing the warnings.
  Solved by adding a header guard for the lines that actually need
  to be reapplied.
- lib/vauth/digest: fix `-Wunreachable-code-break` [6].
- lib/vssh/libssh2: fix `-Wunreachable-code-break` and delete redundant
  block.
- lib/vtls/sectransp: fix `-Wunreachable-code-break` [7].
- lib/vtls/sectransp: suppress `-Wunreachable-code`.
  Detected in `else` branches of dynamic feature checks, with results
  known at compile-time, e.g.
  ```c
  if(SecCertificateCopySubjectSummary)  /* -> true */
  ```
  Likely fixable as a separate micro-project, but given SecureTransport
  is deprecated anyway, let's just silence these locally.
- src/tool_help: delete duplicate declaration for `helptext`.
- src/tool_xattr: fix `-Wunreachable-code`.
- tests: delete duplicate declaration for `unitfail` [8].
- tests: delete duplicate declaration for `strncasecompare`.
- tests/libtest: delete duplicate declaration for `gethostname`.
  Originally added in 687df5c8c3
  (2010-08-02).
  Got complicated later: c49e9683b8
  If there are still systems around with warnings, we may restore the
  prototype, but limited for those systems.
- tests/lib2305: delete duplicate declaration for
  `libtest_debug_config`.
- tests/h2-download: fix `-Wunreachable-code-break`.

[1] a70edb08e9/cmake/PickyWarningsC.cmake
[2] https://ci.appveyor.com/project/curlorg/curl/builds/48553586/job/3qkgjauiqla5fj45?fullLog=true#L1675
[3] https://github.com/curl/curl/actions/runs/6880886309/job/18716044703?pr=12331#step:7:72
    https://github.com/curl/curl/actions/runs/6883016087/job/18722707368?pr=12331#step:7:109
[4] https://ci.appveyor.com/project/curlorg/curl/builds/48555101/job/9g15qkrriklpf1ut#L204
[5] https://ci.appveyor.com/project/curlorg/curl/builds/48555101/job/9g15qkrriklpf1ut#L218
[6] https://github.com/curl/curl/actions/runs/6880886309/job/18716042927?pr=12331#step:7:290
[7] https://github.com/curl/curl/actions/runs/6891484996/job/18746659406?pr=12331#step:9:1193
[8] https://github.com/curl/curl/actions/runs/6882803986/job/18722082562?pr=12331#step:33:1870

Closes #12331
2023-11-21 16:35:42 +00:00