Incremental compilation of Node.js is slow. Currently on a powerful
Linux machine, it takes about 9 seconds and 830 MB of memory to compile
`gen/node_javascript.cc` with g++. This is the longest step when
recompiling a small change to a Javascript file.
`gen/node_javascript.cc` contains a lot of large binary literals of our
Javascript source code. It is well-known that embedding large binary
literals as C/C++ arrays is slow. One workaround is to include the data
as string literals instead. This is particularly nice for the Javascript
included via js2c, which look better as string literals anyway.
Add a build flag `NODE_JS2C_USE_STRING_LITERALS` to js2c. When this flag
is set, we emit string literals instead of array literals, i.e.:
```c++
// old: static const uint8_t X[] = { ... };
static const uint8_t *X = R"JS2C1b732aee(...)JS2C1b732aee";
// old: static const uint16_t Y[] = { ... };
static const uint16_t *Y = uR"JS2C1b732aee(...)JS2C1b732aee";
```
This requires some modest refactoring in order to deal with the flag
being on or off, but the new code itself is actually shorter.
I only enabled the new flag on Linux/macOS, since those are systems that
I have available for testing. On my Linux system with gcc, it speeds up
compilation by 5.5s (9.0s -> 3.5s). On my Mac system with clang, it
speeds up compilation by 2.2s (3.7s -> 1.5s). (I don't think this flag
will work with MSVC, but it'd probably speed up clang on windows.)
The long-term goal here is probably to allow this to occur incrementally
per Javascript file & in parallel, to avoid recompiling all of
`gen/node_javascript.cc`. Unfortunately the necessary gyp incantations
seem impossible (or at least, far beyond me). Anyway, a 60% speedup is a
nice enough win.
Refs: https://github.com/nodejs/node/issues/47984
PR-URL: https://github.com/nodejs/node/pull/48160
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/48502
Reviewed-By: Debadree Chatterjee <debadree333@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Richard Lau <rlau@redhat.com>
For cross-platform compatibility use perl instead of sed.
Fixes: https://github.com/nodejs/node/issues/48496
PR-URL: https://github.com/nodejs/node/pull/48499
Reviewed-By: Marco Ippolito <marcoippolito54@gmail.com>
Reviewed-By: Richard Lau <rlau@redhat.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Mohammed Keyvanzadeh <mohammadkeyvanzade94@gmail.com>
Reviewed-By: Mestery <mestery@protonmail.com>
Reviewed-By: Rafael Gonzaga <rafael.nunu@hotmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/48377
Reviewed-By: Rafael Gonzaga <rafael.nunu@hotmail.com>
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: Paolo Insogna <paolo@cowtech.it>
This is the certdata.txt[0] from NSS 3.90, released on 2023-04-06.
This is the version of NSS that shipped in Firefox 113 on
2023-05-09.
Certificates added:
- BJCA Global Root CA1
- BJCA Global Root CA2
[0] https://hg.mozilla.org/projects/nss/raw-file/NSS_3_90_RTM/lib/ckfw/builtins/certdata.txt
PR-URL: https://github.com/nodejs/node/pull/48416
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Mohammed Keyvanzadeh <mohammadkeyvanzade94@gmail.com>
Reviewed-By: Marco Ippolito <marcoippolito54@gmail.com>
- Run the embedder entry point directly through
runEmbedderEntryPoint(), instead of going through another
JS -> C++ trip through the function returned by
getEmbedderEntryFunction()
- For --build-snapshot, read the snapshot script code directly in C++
and pass it to SnapshotBuilder::Generate(), this makes the entry point
more explicit instead of hiding it in JS land, and also makes it
possible to invoke SnapshotBuilder::Generate() internally to create
a custom snapshot.
- Previously we used process.execPath for the embedder to create
__filename and __dirname in the snapshot builder script while using
process.argv[1] for --build-snapshot (where it's always set) which
results in inconsistencies. We now require the embedder to also set
args[1] when creating the Environment if they intend to run snapshot
scripts with a context that contains __filename and __dirname, which
would be derived from args[1]. If they prefer not to include
build-time paths in the snapshot, we now provide
node::GetAnonymousMainPath() as an alternative.
PR-URL: https://github.com/nodejs/node/pull/48242
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: James M Snell <jasnell@gmail.com>
Run a subset of `tools/dep_updaters/update-eslint.sh` to update
`eslint-plugin-jsdoc` to get rid of a spurious ENOENT error when
running the linter.
PR-URL: https://github.com/nodejs/node/pull/48393
Fixes: https://github.com/nodejs/node/issues/48374
Reviewed-By: Debadree Chatterjee <debadree333@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Mohammed Keyvanzadeh <mohammadkeyvanzade94@gmail.com>
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
PR-URL: https://github.com/nodejs/node/pull/48328
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: Mestery <mestery@protonmail.com>
Reviewed-By: Mohammed Keyvanzadeh <mohammadkeyvanzade94@gmail.com>
The file is checked into git. Ignoring it causes a very non-obvious way
of breaking tarball builds:
1. Download and unpack tarball
2. Check the sources into git with `git init; git add .; git commit -a`
3. Clean the source tree with `git clean -dfx`
4. Run `./configure && make`
5. Observe build failure because config.h is missing
Fixes: https://github.com/nodejs/node/issues/47638
PR-URL: https://github.com/nodejs/node/pull/48174
Reviewed-By: Michaël Zasso <targos@protonmail.com>
Reviewed-By: Richard Lau <rlau@redhat.com>
Reviewed-By: Marco Ippolito <marcoippolito54@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
By default, there is no sha256sum command in macOS.
PR-URL: https://github.com/nodejs/node/pull/48229
Reviewed-By: Marco Ippolito <marcoippolito54@gmail.com>
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Reviewed-By: Mestery <mestery@protonmail.com>
Reviewed-By: Yongsheng Zhang <zyszys98@gmail.com>
Reviewed-By: Tobias Nießen <tniessen@tnie.de>
Reviewed-By: Rafael Gonzaga <rafael.nunu@hotmail.com>
Reviewed-By: Darshan Sen <raisinten@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Avoid repetition by defining a new GYP target.
PR-URL: https://github.com/nodejs/node/pull/47364
Reviewed-By: Richard Lau <rlau@redhat.com>
Reviewed-By: Jiawen Geng <technicalcute@gmail.com>
Reviewed-By: Tobias Nießen <tniessen@tnie.de>
Zlib rarely gets new tags or releases, so now we use the latest commit
on the upstream default branch to check if an update is available.
Refs: https://github.com/nodejs/security-wg/issues/973
PR-URL: https://github.com/nodejs/node/pull/48054
Reviewed-By: Marco Ippolito <marcoippolito54@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
This makes it easier to use third-party dependencies in this tool
(e.g. adding compression using algorithms not available in Python).
It is also much faster - locally js2c.py takes ~1.5s to generate the
output whereas this version takes ~0.1s - and consumes less memory
(~110MB v.s. 66MB).
This also modifies the js2c.py a bit to simplify the output, making
it easier to compare with one generated by the C++ version. Locally
the output from the two are identical. We'll remove js2c.py in a
subsequent commit when the C++ version is used by default.
PR-URL: https://github.com/nodejs/node/pull/46997
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
The update scripts that install dependencies from npm blindly assume
that `npm install` will always install the version that was previously
fetched as `dist-tags.latest`. That is not necessarily true, so instead
always install the specific version that the script thinks is current.
PR-URL: https://github.com/nodejs/node/pull/48101
Reviewed-By: Michaël Zasso <targos@protonmail.com>
Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
Reviewed-By: Darshan Sen <raisinten@gmail.com>
Reviewed-By: Marco Ippolito <marcoippolito54@gmail.com>
Reviewed-By: Mestery <mestery@protonmail.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Cherry-pick 12c8b4d15471cb6211b39c3a2ca5b10fa4b9f12b
Original commit message:
This commit is a suggestion for adding a rule for NULL usages in the
code base. This will currently report a number of errors which could be
ignored using // NOLINT (readability/null_usage)
PR-URL: https://github.com/nodejs/node/pull/17373
Reviewed-By: Jon Moss <me@jonathanmoss.me>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Timothy Gu <timothygu99@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Michael Dawson <michael_dawson@ca.ibm.com>
Reviewed-By: Sakthipriyan Vairamani <thechargingvolcano@gmail.com>
Reviewed-By: Tobias Nießen <tniessen@tnie.de>
Refs: 12c8b4d154
Cherry-pick fc81e801913de3e3f3c0c8e26c105f983a74e539
Original commit message:
Update cpplint.py to check for inline headers when the corresponding
header is already included.
PR-URL: https://github.com/nodejs/node/pull/21521
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: James M Snell <jasnell@gmail.com>
Refs: fc81e80191
Cherry-pick cbc3dd997eb90d629d1b9912b7a5a40eb82343df
Original commit message:
src, tools: add check for left leaning pointers
This commit adds a rule to cpplint to check that pointers in the code
base lean to the left and not right, and also fixes the violations
reported.
PR-URL: https://github.com/nodejs/node/pull/21010
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
Reviewed-By: James M Snell <jasnell@gmail.com>
Refs: cbc3dd997e
Cherry-pick 902998190a55d6915b881936f6dd5b6e9cca6ad8
Original commit message:
tools: fix cpplint.py header rules
THIS COMMIT SHOULD GO WITH THE NEXT. IT WILL FIND NEW LINT.
PR-URL: https://github.com/nodejs/node/pull/26306
Reviewed-By: Gireesh Punathil <gpunathi@in.ibm.com>
Refs: 902998190a
Cherry-pick 0a25ace9c35b62ece4d32fd90b326d8063265109
Original commit message:
tools: move cpplint configuration to .cpplint
PR-URL: https://github.com/nodejs/node/pull/27098
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Reviewed-By: Daniel Bevenius <daniel.bevenius@gmail.com>
Refs: 0a25ace9c3
Cherry-pick afa9a7206c26a29a2af226696c145c924a6d3754
Original commit message:
tools: refloat update link to google styleguide for cpplint
This commit updates two old links to Google's C++ styleguide which
currently result in a 404 when accessed.
PR-URL: https://github.com/nodejs/node/pull/30876
Reviewed-By: Michaël Zasso <targos@protonmail.com>
Reviewed-By: David Carlier <devnexen@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Richard Lau <riclau@uk.ibm.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>
Refs: afa9a7206c
Cherry-pick e23bf8f771aa0bd60e25ff079985fc29b5846403
Original commit message:
tools,src: refloat forbid usage of v8::Persistent
`v8::Persistent` comes with the surprising catch that it requires
manual cleanup. `v8::Global` doesn’t, making it easier to use,
and additionally provides move semantics. New code should always
use `v8::Global`.
PR-URL: https://github.com/nodejs/node/pull/31018
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Richard Lau <riclau@uk.ibm.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: David Carlier <devnexen@gmail.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>
Reviewed-By: Gus Caplan <me@gus.host>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: Stephen Belanger <admin@stephenbelanger.com>
PR-URL: https://github.com/nodejs/node/pull/35569
Reviewed-By: Richard Lau <rlau@redhat.com>
Reviewed-By: Daijiro Wachi <daijiro.wachi@gmail.com>
Reviewed-By: Jiawen Geng <technicalcute@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/35719
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/35866
PR-URL: https://github.com/nodejs/node/pull/36213
Reviewed-By: Franziska Hinkelmann <franziska.hinkelmann@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/36235
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/36324
Reviewed-By: Beth Griggs <bgriggs@redhat.com>
PR-URL: https://github.com/nodejs/node/pull/38851
Reviewed-By: Khaidi Chu <i@2333.moe>
PR-URL: https://github.com/nodejs/node/pull/42416
Reviewed-By: Mestery <mestery@protonmail.com>
Reviewed-By: Darshan Sen <raisinten@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/48098
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/48060
Reviewed-By: Ruy Adorno <ruyadorno@google.com>
Reviewed-By: Richard Lau <rlau@redhat.com>
Reviewed-By: Deokjin Kim <deokjin81.kim@gmail.com>
Reviewed-By: Darshan Sen <raisinten@gmail.com>
Reviewed-By: Mestery <mestery@protonmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Use the `DEPS_DIR` variable to build the path instead of hardcoding it.
PR-URL: https://github.com/nodejs/node/pull/48089
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: Tobias Nießen <tniessen@tnie.de>
Reviewed-By: Marco Ippolito <marcoippolito54@gmail.com>
Currently we use external strings for internalized builtin source code.
However when a snapshot is taken, any external string whose resource
is not registered is flattened into a SeqString (see ref). The
result is that module source code stored in the snapshot does not
use external strings after deserialization. This patch registers an
external string resource for each internalized builtin's source. The
savings are substantial: ~1.9 MB of heap memory per isolate, or ~44%
of an otherwise empty isolate's heap usage:
```console
$ node --expose-gc -p 'gc(),process.memoryUsage().heapUsed'
4190968
$ ./node --expose-gc -p 'gc(),process.memoryUsage().heapUsed'
2327536
```
The savings can be even higher for user snapshots which may include
more internal modules.
The existing UnionBytes implementation was ill-suited, because it only
created an external string resource when ToStringChecked was called,
but we need to register the external string resources before the
isolate even exists. We change UnionBytes to no longer own the data,
and shift ownership of the data to a new external resource class
called StaticExternalByteResource. StaticExternalByteResource are
either statically allocated (for internalized builtin code) or owned
by the static `externalized_builtin_sources` map, so they will only be
destructed when static resources are destructed. We change JS2C to emit
statements to register a string resource for each internalized builtin.
Refs: d2c8fbe9cc/src/snapshot/serializer.cc (L633)
PR-URL: https://github.com/nodejs/node/pull/47055
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
