To simplify the directory layout.
- OS400 and vms support move from `packages` to `projects`.
- Windows README and `generate.bat` files move from `projects`
to `projects/Windows`.
Closes#20271
To run checksrc and spacecheck on the source tree. Also for cmake
to sync up with autotools' `checksrc` target.
- cmake: `curl-lint`
With `-DCURL_LINT=ON`, checks run automatically for all targets.
- autotools: `lint`
Closes#20175
Also:
- memdebug: fix symbol collision in unity builds.
- memdebug: fix compiler warning by making a variable static.
Follow-up to c77bed81a2#19657Closes#19666
Replacing a combination of custom logic in the main script and relying
on CMake's built-in Find module, with code and behavior used for
the rest of dependencies.
Also to:
- add version detection in the non-pkg-config path.
- make `GNUTLS_INCLUDE_DIR` and `GNUTLS_LIBRARY` take precedence over
pkg-config. As with other dependencies.
- document the above two configuration options.
- prepare for #16973, which originally introduced this local Find
module.
The local module is doing largely the same as CMake's built-in
FindGnuTLS. Differences:
- honors `CURL_USE_PKGCONFIG`.
- returns GnuTLS version for non-pkg-config detection.
- consistently returns `GNUTLS_VERSION`.
(CMake's built-in uses s different name in <3.16.)
- CMake 3.16+ returns an imported target. curl supports 3.7,
therefore we may only use it conditionally, which isn't worth it.
Cherry-picked from #16973Closes#19163
The implementation was incomplete and lesser than the other backends. No
one ever reported a bug or requested enhancements for this, indicating
that this backend was never used.
Closes#18700
Also:
- GHA/linux, macos: test `install` with autotools too.
- GHA/linux, macos: enable fish and zsh completion in an autotools job.
- scripts: drop dynamic perl checks redundant after this patch.
- scripts: drop two interim variables.
- autotools: make `build-certs` target check perl first.
- autotools: replace `$(PERL)` with `@PERL@` to match the pattern used
in most automake scripts. For consistency. This makes `PERL` be
defined at configure-time, as opposed to make-time, for these cases
now.
Closes#18141
`checksrc.bat` was outdated and required Perl for `checksrc.pl` anyway.
Rewrite `checksrc-all.sh` in Perl, making it usable in envs without
a POSIX shell.
Closes#17882
This is a script for building OpenSSL to be used with legacy Visual
Studio builds.
I don't think it is our job to provide nor maintain OpenSSL build
scripts.
Remove
Closes#17879
These libraries do not support TLS 1.3 and have been marked for removal
for over a year. We want to help users select a TLS dependency that is
future-proof and reliable, and not supporting TLS 1.3 in 2025 does not
infer confidence. Users who build libcurl are likely to be served better
and get something more future-proof with a TLS library that supports
1.3.
Closes#16677
1. 'make clean' in the root dir now also invokes 'make clean' in the
tests subdir so that it cleans up better recursively. The Makefile.am
does not list 'tests' as a normal subdir to avoid building that
directory for a normal make invoke.
2. 'make clean' in the tests/libtest and tests/unit directories now
*explicitly* remove the unity build executables even if this is not a
unit build. This, because those files may be leftovers from previous
builds and such leftovers can otherwise linger around and since
'runtests.pl' dynamically acts differently based on the mere *presence*
of those files, they can keep you fooled for a while until you
realize...
Closes#16986
TL;DR: Save 10 minutes of CI time for GHA/macos jobs using pre-fills and
add pre-fill verification for Apple and Windows. Also restores Xcode job
and saves 1.5-10 minutes configuring iOS jobs.
Pre-filling feature detection results can bring down the CMake configure
step to ~5 seconds on most GHA runners, ~10 seconds in slow envs like
Cygwin/MSYS2.
The potential savings per job are:
- 5-40 (average 19) seconds on GHA/macos (33 jobs)
- ~10 seconds on GHA for iOS GNU Makefile (1 job)
- 1.5-10 minutes on GHA for iOS Xcode generator (1 job)
- 10 seconds on GHA/linux with native Ubuntu (12 jobs)
- 40 seconds for Cygwin/MSYS2 (2 jobs)
- 5-10 seconds for virtualized BSDs, native CPU (3 jobs)
- ~60 seconds for virtualized BSDs, emulated CPU (1 job)
On native Windows pre-filling has been in place for a long time and
saving 8 minutes (VS2019-VS2015) to 1.5-2 minutes (VS2022), 3 minutes
(VS2022 UWP), and 30-60 seconds (MinGW), per CI job.
The downside is that detection results need to be manually collected and
filtered to those that universally apply to all platforms that they are
enabled on. Another downside is that by using a cache, we're not running
the actual detections, and thus won't catch regressions in them. It
means we must make sure that the cache is solid and matches with actual
detections results. An upside is that it gives a rough overview of which
features are available on which platforms. Another upside is pre-filled
values do work for feature detections skipped for cross-builds, e.g.
`HAVE_WRITABLE_ARGV`.
This PR adds a pre-fill cache that supports all Unixes (except OmniOS)
used in CI, and makes it usable with an internal option. It also enables
it for GHA/macos CI jobs, where the maximum savings are. And also for
the two iOS [1] and two Cygwin/MSYS2 jobs. The latters don't have
pre-fill checks and we can drop them if they turn into a hassle.
Saving:
- 10 minutes of CI time per GHA/macos workflow run. [2]
- ~80 seconds per GHA/windows workflow run with Cygwin/MSYS2.
(offsetting the cost of pre-fill verifications)
- 1.5-10 minutes per GHA/non-native runs with iOS jobs. [3]
You can enable pre-fill locally with `-D_CURL_PREFILL=ON`. It's
experimental, and if you experience a problem, file a PR or an Issue.
This PR also adds a pre-fill checker for macOS and MinGW/MSVC Windows
GHA jobs to catch if the cache diverges from real detections. It also
adds this logic to AppVeyor, but doesn't enable it due to the perf
penalty of 2 minutes mininum.
The pre-fill checker works by configuring out-of-tree with and without
pre-fill, then diffing their `lib/curl_config.h` outputs.
Exceptions are 3 detection results exposed indirectly [4], and missing
to expose 2, of which one is the C89 header `stddef.h`. While we assume
the C99 `stdint.h` available outside iOS. We can expose them in the
future, if necessary.
The pre-fill checks cost in total:
- ~20 seconds for macOS
- ~40 seconds for MinGW on GHA
- ~80 seconds for MSVC on GHA (UWP would be 2x this)
An extra time saving potential is caching type sizes. They are
well-known, and seldom change, esp. in CI. GHA/Windows jobs spend 8-17
seconds per job on these ~12 feature checks. ~5s on Cygwin/MSYS2. Couple
of seconds on other platforms. (This PR doesn't make this optimization.)
Another opportunity is doing the same for autotools, which typically
spends more time in the configuration step than cmake.
[1] Xcode job restored as a
follow-up to be5f20202c#16302
[2] GHA/macos cmake configure times in seconds:
Job | Bef. | After | Gain
:----------------------------------------------- | ----: | ----: | ----:
CM clang GnuTLS !ldap krb5 | 21.2 | 4.5 | 16.7
CM clang LibreSSL !ldap heimdal c-ares +examples | 13.3 | 3.9 | 9.4
CM clang OpenSSL +static libssh +examples | 20.0 | 4.6 | 15.4
CM clang OpenSSL IDN clang-tidy~ (w/chkprefill) | 15.7 | 18.6 | -2.9
CM clang OpenSSL gsasl rtmp AppleIDN | 25.0 | 4.7 | 20.3
CM clang OpenSSL torture !FTP | 15.3 | 4.5 | 10.8
CM clang OpenSSL torture FTP | 25.0 | 5.9 | 19.1
CM clang SecureTransport debug | 18.0 | 3.8 | 14.2
CM clang macos-13 SecureTransport | 45.8 | 12.4 | 33.4
CM clang macos-14 SecureTransport | 15.8 | 4.6 | 11.2
CM clang macos-15 SecureTransport | 26.8 | 6.1 | 20.7
CM clang mbedTLS openldap brotli zstd | 15.1 | 6.5 | 8.6
CM clang wolfSSL !ldap brotli zstd | 27.0 | 4.4 | 22.6
CM gcc-12 GnuTLS !ldap krb5 | 39.1 | 8.7 | 30.4
CM gcc-12 LibreSSL !ldap heimdal c-ares +examples| 23.8 | 7.2 | 16.6
CM gcc-12 OpenSSL +static libssh +examples | 20.7 | 8.5 | 12.2
CM gcc-12 OpenSSL gsasl rtmp AppleIDN | 23.1 | 10.1 | 13.0
CM gcc-12 SecureTransport debug | 21.1 | 4.8 | 16.3
CM gcc-12 mbedTLS openldap brotli zstd | 21.4 | 5.8 | 15.6
CM gcc-12 wolfSSL !ldap brotli zstd | 21.1 | 6.9 | 14.2
CM gcc-14 macos-13 SecureTransport | 61.9 | 18.7 | 43.2
CM gcc-14 macos-14 SecureTransport | 30.5 | 6.4 | 24.1
CM gcc-14 macos-15 SecureTransport | 32.7 | 8.4 | 24.3
CM llvm@15 GnuTLS !ldap krb5 | 21.1 | 7.5 | 13.6
CM llvm@15 LibreSSL !ldap heimdal c-ares +exampl~| 24.6 | 6.8 | 17.8
CM llvm@15 OpenSSL +static libssh +examples | 19.0 | 6.4 | 12.6
CM llvm@15 OpenSSL gsasl rtmp AppleIDN | 19.0 | 8.2 | 10.8
CM llvm@15 SecureTransport debug | 18.0 | 5.4 | 12.6
CM llvm@15 macos-13 SecureTransport | 66.2 | 25.7 | 40.5
CM llvm@15 macos-14 SecureTransport | 31.9 | 6.1 | 25.8
CM llvm@15 mbedTLS openldap brotli zstd | 19.5 | 8.9 | 10.6
CM llvm@15 wolfSSL !ldap brotli zstd | 24.3 | 5.9 | 18.4
CM llvm@18 macos-15 SecureTransport | 33.8 | 6.4 | 27.4
Total | 856.8 | 257.3 | 599.5
Before: https://github.com/curl/curl/actions/runs/13311042735/job/37173478424
After: https://github.com/curl/curl/actions/runs/13313927119/job/37183206426?pr=15841
[3] iOS:
Before: https://github.com/curl/curl/actions/runs/13326401704?pr=15841
After: https://github.com/curl/curl/actions/runs/13332177764?pr=15841
[4] detection results exposed indirectly in `curl_config.h`:
- `HAVE_FILE_OFFSET_BITS` via `_FILE_OFFSET_BITS`
- `HAVE_GETHOSTBYNAME_R_*_REENTRANT` via `NEED_REENTRANT`
- `HAVE_SOCKADDR_IN6_SIN6_ADDR` via `USE_IPV6`
Closes#15841
Add CMake test project consuming curl via these methods:
`FetchContent`, `add_subdirectory()`, `find_package()`.
Also:
- GHA/distcheck: run these tests in CI.
- cmakelint: exclude a warning for calling "wonky-cased" built-in
CMake functions, such as `FetchContent_Declare()`.
Closes#16126
It had shorthand aliases to launch `./configure` and
`./configure --with-openssl`. The former hasn't worked for a long while
because of missing TLS.
Its `ca-bundle` and `ca-firefox` targets have been broken for 2.5 years
till recently. These targets also exist in `./configure` and have been
working all along.
Also:
- cmake: add support `curl-ca-bundle` and `curl-ca-firefox` targets.
- tests/testcurl.pl: drop obsolete build logic.
Closes#16094
Rework the way `tool_hugehelp.c` is included in builds.
After this patch, with `./configure` and CMake `tool_hugehelp.c` is only
compiled when building with manuals enabled. With manuals disabled this
source file is not used anymore. The method is similar to how
8a3740bc8e implemented `tool_ca_embed.c`.
`./configure` always generates it as before, otherwise the build fails.
- winbuild: rework to not need `buildconf.bat`, but automatically use
`tool_hugehelp.c` if present (e.g. when building from an official
source tarball) and enable `USE_MANUAL` accordingly.
- `buildconf.bat`: after dropping `tool_hugehelp.c` generation, the only
logic left was `cp Makefile.dist Makefile`. This allowed to launch
winbuild builds via GNU Make in a Git repo. Drop this option together
with the batch file.
- build `libcurltool` without `USE_MANUAL` macro to exclude the manual
and the dependence on the generator commands. Drop relying on
`UNITTESTS` for this purpose.
Follow-up to 96843f4ef7#16068
- `src/mkhelp.pl`: include `tool_hugehelp.h` before using `USE_MANUAL`
to have it set in `config-*.h` builds with source tarballs created
with manual but without zlib.
Closes#16081
The new detection method also allows to enable librtmp without using
OpenSSL as a curl TLS backend at the same time.
Also:
- implement manual version detection for librtmp.
Version info is in hex. With CMake 3.13 and newer, extract it as a hex
number. With earlier CMake version, just strip the leading zeroes.
Doing more here seems overkill because librtmp has been standing
at 2.3/2.4 for a decade now. Bumping into hex digits seems unlikely
before deprecating CMake 3.13 support.
librtmp advertises v2.4 via its `pkg-config` module, and v2.3 via
its public header. The latter shows up in `curl -V` and either can
be shown at configure-time depending on detection method.
This isn't a curl bug.
- GHA/macos: enable rtmp in a job.
- apply the "half-detection" fix to the Find module.
`librtmp` is also affected (in CI too), because it depends on libssl and
libcrypto.
Closes#15832
- Remove wolfSSL from the legacy projects for Visual Studio 2010 - 2013.
It's no longer possible to maintain the custom build configuration of
wolfSSL for these old versions of Visual Studio.
Note support for wolfSSL was recently added to the winbuild build system
in 4de627ab and the user could possibly make their own wolfSSL build and
attempt to link it using an old version of Visual Studio that way.
Ref: https://gist.github.com/jay/5f6d8d5ba15c12c7457e3216a94da72d
Closes https://github.com/curl/curl/pull/15468
Move LDAP detection to its own Find module.
It supports `pkg-config` and the standard detection method used for
other dependencies, with version detection.
In curl CI it fixes LDAP detection in the OpenBSD job.
Closes#15273
It enables running pytests in cmake jobs, regardless of underlying build
tool choice (= makes it work with ninja.)
Also:
- drop pytest logic launching `make` and exiting in case of failure.
Maybe there is a better way and keep this functionality somehow, bind
it to a command-line option? make it fail softly?
- GHA/linux: invoke pytest via the build, not directly.
- autotools: add missing dummy runtests targets when cross-compiling.
Closes#15034
- cmake: drop `configure.os`.
This also includes OS version, but thus far it's not important enough
to include it.
- autotools: drop redundant, autotools-only `{target|host}.vendor`.
(it's part of the triplet in `{target|host}`.)
- swap order to `*.cpu` -> `*.os` to match triplet-order.
- cmake: drop redundant `target`.
It's manually filled and only in a (so far) few CI jobs. Let's revisit
when this becomes useful.
- move `buildinfo.txt` to build root.
- dist: add `buildinfo.txt` to `DISTCLEANFILES`.
- autotools: detect human readable compiler version.
- autotools: replace `XXYY` `compiler.version` with "X.Y"-style.
(also to match cmake.)
- autotools: use distinct `compiler_id` for Apple clang: `APPLECLANG`.
To match cmake and also because the the "X.Y"-style version number
is the Apple version, while `XXYY` was a value roughly translated to
mainline llvm/clang version.
- show buildinfo at the end of the configure stage, when run in CI, or
when `CURL_BUILDINFO` or `CURL_CI` env is set.
Follow-up to 1fdea16846#14802
Assisted-by: Dan Fandrich
Ref: https://github.com/curl/curl/pull/14802#issuecomment-2334942991Closes#14822
Some of the `.in` files were listed in `EXTRA_DIST`. Delete them.
`.in` files (passed to `AC_CONFIG_FILES`) are added automatically
to the distro by autotools.
Closes#14821
For: libgsasl, libidn2, libssh, libuv.
The new Find modules retain using `pkg-config` natively, not as a "hint"
for the CMake-native detection. Of the pre-existing Find modules, only
FindNettle, and FindGSS (with customized code) work this way. Align
detection code for the new modules and add version detection for the
CMake-native paths.
Also, add CMake-native detection for `libgsasl`.
The remaining outlier in `CMakeLists.txt` is GnuTLS, which has
a CMake built-in Find module, but which lacks `pkg-config` support,
required for vcpkg. It remains unchanged.
Another part-outlier is `libssh`, which keeps requiring the trick
`find_package(libssh CONFIG QUIET)` for reasons I could not yet figure
out.
Closes#14555
- `FindCARES` -> `FindCares`
- `FindLibPSL` -> `FindLibpsl`
- `FindLibSSH2` -> `FindLibssh2`
- `FindQUICHE` -> `FindQuiche`
- `Findrustls` -> `FindRustls`
Our convention for naming Find modules (the part after the `Find`
prefix, also called as 'package name') is:
Always start with uppercase. Follow with lowercase, unless there is
a clear preference for a stylized name. E.g. the project itself uses it
that way with a matching `<Name>Config.cmake` file, or we use it that
way elsewhere, or the name is an acronym.
Ref: #14580Closes#14601
Enable with CMake option `-DCURL_USE_WOLFSSH=ON`. Customize with
`-DWOLFSSH_INCLUDE_DIR=<path-to-wolfssh>/include`,
`-DWOLFSSH_LIBRARY=<path-to-wolfssh>/lib/libwolfssh.a`.
It requires the wolfSSL TLS backend.
Closes#14568
The documented and mandated step has been to not use buildconf but to
invoke 'autoreconf -fi' for four years already.
This change only drops buildconf from the release tarball, it remains
present in git for now.
Follow-up to 85868537d6Closes#14412
Use these words and casing more consistently across text, comments and
one curl tool output:
AIX, ALPN, ANSI, BSD, Cygwin, Darwin, FreeBSD, GitHub, HP-UX, Linux,
macOS, MS-DOS, MSYS, MinGW, NTLM, POSIX, Solaris, UNIX, Unix, Unicode,
WINE, WebDAV, Win32, winbind, WinIDN, Windows, Windows CE, Winsock.
Mostly OS names and a few more.
Also a couple of other minor text fixups.
Closes#14360
Also fix the .dist replacing by avoiding all Makefiles because it
otherwise also went into the temporary release folder and got confused
about the Makefile.dist in there.
Make it possible to rebuild an identical copy from a release tarball. It
was previously only possible from a checked out git repository.
- add release-tools.sh to dist
- keep Makefile.dist around to include it in dist
- regenerate tool_huge.c with the new version in dist
- fix the dist CI job to not do make clean like before
Closes#14336
- scripts/log2changes.pl was not included in release tarballs, which broke
reproducible builds
- since log2changes uses git to generate the contents, it makes it difficult
to generate the same contents later (it would need to be fixed)
- the CHANGES file has outlived its purpose. the main changes are in the
RELEASE-NOTES, the rest are better tracked directly using git or on GitHub
- put a fixed CHANGES.md in there instead pointing out where the info lives
now
Closes#14331
- add CMake option to verify if the `CMake/*.cmake`, `CMake/*.in` files
are listed as distributable in autotools' `EXTRA_DIST`. The check can
be enabled with `-DENABLE_DIST_TEST=ON` CMake option.
- add CI job to that effect.
Ref: #14320Closes#14323
- prefer `.bat` extension over `.cmd` for MS-DOS batch, which also
avoids confusion with OS/400 `.cmd` files.
- cleanup `echo` quotes, drop them consistently.
- delete empty output line from one of the error branches.
- prefer lowercase commands like the rest of MS-DOS batches.
- delete a contraction.
- drop backticks from error message.
- use `nmake.exe` consistently.
- use equal/not-equal operator style consistently.
- inline a single-line `if` branch.
- delete exceptions and rules dealing with Windows `.cmd` extension.
Closes#14084
