This release is mostly hardening against untrusted repository content.
It fixes two checkout issues: a crafted whiteout name that strips to
".." and makes a --whiteouts checkout delete the destination's parent
(GHSA-cp2j-qv7x-wcw4), and union checkouts following a symlink planted
at a directory name and writing outside the destination
(GHSA-7pq7-3pmj-9wp6). Composefs and commit signature checks with the
OpenSSL ed25519/SPKI backends now accept only EVP_DigestVerify() == 1,
where error returns used to count as a valid signature. Static delta,
content header, commit and curl header parsing gain bounds checks.
Behaviour changes: the default min-free-space is now min(3%, 1GB), and
the systemd generator leaves boot.mount to systemd-fstab-generator when
/etc/fstab has a /boot entry.
https://github.com/ostreedev/ostree/releases/tag/v2026.5
AI-Generated: Uses Claude Code (Claude Opus 5.5)
Signed-off-by: Ricardo Salveti <ricardo.salveti@oss.qualcomm.com>
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
The test expects a message sent from another thread 0.2s later to arrive
within a 0.5s poll window, which a loaded test host misses. Upstream
marks it flaky(reruns=3), but pytest-rerunfailures is not available.
Adjust the musl sed to the new end of the -k expression.
AI-Generated: Uses Claude Code
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
test_async_cat_file_ranges fetches a file from a public S3 bucket, which
fails without DNS and internet access on the target, as in sysvinit
qemu images where resolv.conf is empty.
AI-Generated: Uses Claude Code
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
run-ptest wrote each test's output into a file in the image, so an
intermittent gpiod-glib-test failure in testimage left nothing to debug.
Print the output when a test fails.
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
mount/blkdev-fsname-hello{,-ll} attach a loop device, but loop is built
as a module and was not in the ptest image, so losetup failed. Recommend
kernel-module-loop and modprobe it in run-ptest, since it is not loaded
on demand when /dev/loop-control has not been created.
AI-Generated: Uses Claude Code
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
musl defines CLOCK_SGI_CYCLE, so on aarch64 cycle.h uses
clock_gettime(CLOCK_SGI_CYCLE) as its tick counter. Linux returns
EINVAL for that clock, every measurement reads as zero, and
FFTW_MEASURE planning loops forever, which times out the bench ptest.
Skip the SGI clock on Linux.
AI-Generated: Uses Claude Code
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
Upgrade to release 10.17.0:
- Added {class}pikepdf.ObjectSet, which records the objects that a
walk of a PDF's object graph has already visited. Objects in a
PDF are shared and may refer back to their ancestors, so a search
that only limits its depth can take exponential time on quite
ordinary files. ObjectSet.add() reports whether an object is new,
and tells apart direct objects and objects of different Pdf,
which a set of objgen cannot. See {ref}walking-object-graphs.
- Added {meth}pikepdf.Object.is_same_object_as, which tests whether
two references lead to the same object. Neither is, == nor
comparing objgen answers this reliably.
- Added {meth}pikepdf.Pdf.content_scopes and
{meth}pikepdf.Page.content_scopes, which iterate over every
content stream that pages can draw, each with the resource
dictionary it uses: the page itself, form XObjects, annotation
appearance streams, tiling patterns, Type 3 glyph procedures and
soft masks, however they are nested or shared. This is the way to
find every font, image or colour space a document uses. Each
{class}pikepdf.ContentScope is reported once for the document;
pages= selects pages and kinds= selects what is reported.
- Added {meth}pikepdf.Page.get_resources, which returns the
resources of a page or form XObject for reading. Unlike
{attr}pikepdf.Page.resources, it finds resources inherited from
the page tree, does not create an empty dictionary when there are
none, and returns None instead of raising when /Resources is
malformed.
- Fixed comparing two objects with == taking exponential time when
the objects being compared share children, such as the same file
opened twice. Each pair of objects is now compared once.
- Fixed {meth}pikepdf.Pdf.add_pages_from hanging on a page with an
action whose /Next actions are shared or form a loop.
- Fixed {meth}pikepdf.Pdf.open_outline reading only the first item
at each level of an outline whose items are direct objects. Such
items are now made indirect when the outline is saved, as the PDF
specification requires.
- Reading a stream that will not decode now raises
{exc}~pikepdf.exceptions.DataDecodingError with the reason,
whether the stream's data was read from a file or set from
memory. Previously a stream read from a file raised a
{exc}~pikepdf.exceptions.PdfError that said only "read_bytes
called on unfilterable stream", and the reason was available only
from {meth}pikepdf.Pdf.get_warnings. DataDecodingError is a
PdfError, so existing handlers are unaffected.
- An exception raised by a JBIG2 decoder now reaches the caller of
{meth}pikepdf.PdfImage.as_pil_image,
{meth}pikepdf.PdfImage.extract_to and
{meth}pikepdf.PdfImage.read_bytes unchanged, with its own type,
message and traceback. Previously, for an image read from a file,
it was replaced by a generic DataDecodingError or PdfError.
- Added {exc}~pikepdf.exceptions.QpdfRuntimeError, raised for
errors from qpdf that pikepdf has no more specific exception for.
These were previously raised as a bare RuntimeError, which except
pikepdf.PikepdfError did not catch. The new exception derives
from both {exc}~pikepdf.exceptions.PikepdfError and RuntimeError,
so handlers written for RuntimeError still work.
- The empty name / is now supported. The PDF 1.7 and 2.0
specifications define a lone slash as a valid name made of zero
characters, but pikepdf.Name('/') raised ValueError, and
dictionaries refused / as a key. This broke
{meth}pikepdf.Pdf.add_pages_from on files with an empty-named
destination such as /Dests << / 3 0 R >>. Such names can now be
created, used as dictionary keys, and migrated during page
copying. bool(Name('/')) is False, like an empty
{class}pikepdf.String. Name(''), which lacks the leading slash,
is still rejected. PDF/A does not forbid empty names, and the
PDF/A validator accepts them.
- AcroFormField.get_inheritable_field_value_as_name() now returns a
{class}pikepdf.Name, as documented, instead of a str. When the
value is missing or not a name, it returns the empty name
Name('/'), which is falsy, instead of ''. Comparisons such as ==
'/Tx' are unaffected.
- Fixed an out-of-bounds read when opening a PDF from a Python file
object whose readinto() reports more bytes than were requested.
This now raises ValueError, as writing to such a stream already
did. Non-blocking streams, whose readinto() returns None, now
raise ValueError instead of hanging.
- Fixed building from source on PyPy, which failed to compile since
v10.0. PyPy now reads Python file objects through readinto(),
like CPython, instead of copying the result of read().
- {func}pikepdf.open reads directly from the file descriptor when
given a filename, instead of calling the seek() and readinto()
methods of a Python file object for every read. Opening and
reading a large file with many objects is about four to five
times faster, matching the speed of memory mapping
(access_mode=AccessMode.mmap). Streams passed to
{func}pikepdf.open, including plain file objects, are read
through their methods as before.
- Documented {class}pikepdf.TokenFilter with worked examples that
shorten numbers and convert RGB colors to grayscale, and
explained when to use a token filter instead of
{func}pikepdf.parse_content_stream. The API reference no longer
says that the parser should not be used to edit content streams.
The TokenFilter.handle_token documentation now says correctly
that exceptions it raises propagate to the caller.
Extend RDEPENDS with the missing runtime dependencies
python3-fonttools and python3-jsonschema which are required
to import pikepdf.pdfa.
Install pdfa_differential.py to pass the tests from
test_pdfa_differential.py. Fixes:
E ModuleNotFoundError: No module named 'pdfa_differential'
This work was sponsored by GOVCERT.LU.
Signed-off-by: Leon Anavi <leon.anavi@konsulko.com>
Signed-off-by: Khem Raj <raj.khem@gmail.com>
Upgrade to release 3.10.4:
This release routes every file path, archive, download, URL fetch and
external program through one sandbox that keeps reads and writes inside
the trusted data directories, refuses path traversal, symlink escapes
and decompression bombs, blocks requests to internal network addresses,
and runs Java and other third-party tools only from trusted absolute
locations with validated options and input. Pickled models now load
through a restricted unpickler, XML and JSON parsing are hardened, and
regular expressions, recursion and slow algorithms in the tokenizers,
parsers, stemmers and corpus readers are bounded against
denial-of-service input. Text that NLTK writes to the terminal, to CSV
files, to the WordNet browser and to third-party tool input is sanitized
against injection.
Security changes you may notice
Every refusal below raises with a message that starts with "Security
Violation" and names the check, so a failure is easy to search for (PR
- Data directories (nltk.pathsec, nltk.data): corpora, models and
every other resource are read and written only inside the
directories listed in nltk.data.path. A path that leaves a data
root, a symlink that points out of one, and a NUL byte or encoded
traversal in a resource name are refused. A data root, or a
directory on the way to a resource, that another account can
write to is skipped with a warning naming the directory and the
remedy. Keep private data in a private directory and add it to
nltk.data.path; /tmp is not a data root. Model save and load
methods (perceptron, maxent, named-entity chunker, Weka, crf,
hunpos, Stanford) accept destinations inside a data root only (PR
#3797, #3753, #3850, #3761, #3755, #3759, #3760, #3763, #3806,
#3807, #3808, #3809, #3813, #3814, #3815, #3848, #3851, #3852,
#3853, #3863, #3871, #3888, #3945, #3961, #3969, #3981)
- Downloader: downloaded package archives stay private to the
installing account (0600). An install another account must read
is extracted: a download made as root or outside the installing
account's home extracts on its own, and nltk.download(...,
extract=True) / python -m nltk.downloader --extract (or
--no-extract) decide explicitly. This also unpacks the packages
the index leaves zipped (wordnet, omw-1.4 ...); the extracted
files are readable by every account and are found before the
archive. find() names an archive it cannot read and says how to
extract it instead of reporting it "not found". The downloader
installs only into directories no other account can write to,
treats an archive another account owns or can write to as stale
and downloads it afresh, extracts only from an archive no other
account could have changed since its checksum was verified,
bounds the size of what it fetches, and refuses index entries and
archive members that would write outside the package directory
(PR #3929, #3804, #3850, #3778)
- Pickles: nltk.data.load(..., format="pickle") reconstructs plain
data only; a pickle that names any class or function is refused.
Model loaders that need classes use an explicit allowlist of
audited globals, so a gadget such as os.system can never be
reconstructed from a model file. chat80 shelve values load the
same way (PR #3794, #3754, #3757, #3802, #3817, #3823, #3824)
- XML, JSON and YAML: nltk.xmlsec refuses any document that
declares an entity or references an external one (billion-laughs,
external-entity and DTD attacks) and bounds the tree depth, and
every corpus XML view, the downloader index, Toolbox and
internals parse through it. JSON resources load with a depth and
size bound; YAML (the markdown corpus front matter and
nltk.data.load(..., format="yaml")) loads through a bounded safe
loader. Oversized or too deeply nested input raises ValueError
naming the limit (PR #3754, #3805, #3828, #3933, #3972, #3974)
- Regular expressions (nltk.redos): every regular expression NLTK
compiles, including one you pass to RegexpTokenizer,
RegexpStemmer, RegexpParser, re_show, TokenSearcher.findall,
tgrep, nltk.help tag-set lookups and the concordance and nemo
GUIs, now goes through nltk.redos on the regex engine. A pattern
longer than 100,000 characters, nested more than 100 deep, with
more than 1,000 capturing groups or an excessive repetition
product is refused with nltk.redos.error, a subclass of re.error;
a match that runs longer than nltk.redos.DEFAULT_TIMEOUT (5
seconds) raises TimeoutError instead of hanging. Patterns keep re
syntax; match objects are the regex engine's (PR #3825, #3829,
#3830, #3831, #3832, #3833, #3834, #3835, #3836, #3837, #3838,
#3839, #3840, #3841, #3842, #3843, #3844, #3845, #3846, #3796,
#3801, #3906, #3907, #3908, #3910)
- Regular expressions made linear: the leading-run and re-anchoring
patterns that a timeout cannot catch were rewritten or bounded in
the word, Treebank, blankline, TextTiling, toktok and syllable
tokenizers, the readline of corpus streams, the CoNLL SRL, TIMIT,
senseval, pl196x, NKJP, Alpino, YCOE, IEER, ACE, VerbNet, reviews
and Lin readers, relextract, the valuation parser, the CCG
lexicon parser, dependency grammars and chomsky_normal_form, each
verified on the real corpus (PR #3947, #3934, #3932, #3938,
#3935, #3933, #3869, #3944, #3911, #3909, #3953, #3895)
- Size, depth and time bounds: trees deeper than
nltk.tree.MAX_TREE_DEPTH (500), WordNet and other relation walks
deeper than nltk.util.MAX_RECURSION_DEPTH (500), tgrep deeper
than 50, Toolbox deeper than 500, dependency graphs deeper than
200, are refused with ValueError naming the limit rather than
exhausting the stack, and the stemmers, distance metrics,
skipgrams, parsers and grammar transforms run in linear or
bounded time with identical output for legitimate input.
chart_parse stops after nltk.parse.chart.DEFAULT_MAX_TIME (5
seconds of wall-clock, needed only for feature grammars;
max_time=None disables it). Tree.__repr__ no longer recurses, so
a deep accepted tree prints (PR #3827, #3885, #3895, #3767,
#3784, #3917, #3958, #3960, #3977)
- External tools and Java: a tool binary is resolved from an
absolute path or a trusted search path only, never from the
current directory; the binary, its directory and the model and
data files handed to it must not be writable by another account
(symlinked and multi-call executables such as a packaged Graphviz
dot are supported). java() accepts a main class as the first
argument only (no -jar, @argfile or .java source file), validates
its options against an allowlist, and loads jars only from a data
root unless NLTK_ALLOW_UNSAFE_JARS=1 opts out with a warning.
CoreNLP server options are allowlisted. Stanford, Malt, Senna,
hunpos, Prover9, megam, TADM, Weka and crf wrappers validate
their paths and the text they pipe to the tool against line and
argument injection (PR #3772, #3771, #3975, #3743, #3782, #3788,
#3768, #3798, #3887, #3858, #3950, #3951, #3812, #3816, #3868,
#3894, #3941, #3850, #3925)
- Network: the downloader, nltk.data.load of a URL and the CHILDES
reader fetch through pathsec.urlopen, which refuses addresses
that are not globally routable (loopback, private, link-local and
multicast, including tunnelled and obfuscated forms), pins the
validated address for the connection, allows file:// with a local
authority only, and refuses a proxied fetch unless
NLTK_ALLOW_PROXIED_URLOPEN=1 (or nltk.pathsec.ALLOW_PROXIED_FETCH
= True) asserts the proxy is trusted. The CoreNLP client talks to
its local server with requests and is not affected (PR #3847,
#3810, #3750, #3850)
- Terminal, CSV, DOT and HTML output: every print in the library
goes through nltk.termsec.safe_print, which renders control
characters, escape sequences and unbalanced bidi overrides from
corpus text as visible \xNN escapes instead of letting them reach
the terminal. The CSV files the twitter and sentiment helpers
write neutralise formula-leading and control-sequence cells,
AlignedSent DOT labels are escaped, the WordNet browser escapes
corpus text, and nltk.data.load's verbose output no longer prints
the full resource URL (PR #3914, #3915, #3937, #3946, #3931,
#3930, #3856, #3902)
- Corpus stores and encodings: the PanLex Lite and chat80 sqlite
stores open read-only on a pinned descriptor with their schema
screened and a statement step budget, chat80 shelves are read by
a key manifest, Toolbox SFM markers are sanitised, and a resource
encoding that is not a text codec (bz2_codec and friends) is
refused (PR #3961, #3964, #3959, #3888, #3800, #3962)
- Package contents: the wheel no longer ships nltk.test.unit (the
attack harnesses, advisory probes and their fixtures), so pip
install nltk is about half the size and import nltk.test.unit
works from a source checkout or the sdist only. The doctests
under nltk.test still ship (PR #3954)
Security test and CI infrastructure
- Attack and regression harnesses, advisory probes, the sandbox
conftest fixtures, the pre-commit guards (no unsandboxed open,
every regex through redos, every pickle through picklesec, every
print through safe_print), load-invariant CPU-time timing rules,
CI matrix and runner fixes, pinned third-party downloads, and the
AUTHORS update (PR #3744, #3749, #3762, #3764, #3770, #3776,
#3791, #3795, #3803, #3818, #3820, #3821, #3822, #3849, #3855,
#3860, #3861, #3862, #3864, #3865, #3866, #3876, #3884, #3899,
#3904, #3905, #3912, #3917, #3924, #3926, #3927, #3939, #3940,
#3943, #3948, #3949, #3963, #3968, #3971, #3976, #3982, #3983,
#3984, #3985)
Other changes
- Added RTL language syntactic tree visualization support (PR
#3472)
- Fix#3394: Ensure punkt_tab is downloaded when punkt is requested
(PR #3505)
- Show README.md as the PyPI long description (PR #3746)
- Add stopwords.words("english-extended") (PR #3747)
- Krippendorff's alpha: treat placeholder labels as missing data
(#2865, #2732) (PR #3752)
- Fix shared defaults in chat80 Concept (PR #3773)
- Fix Java wrappers decoding str output from java()
(universal_newlines) (PR #3777)
- Prevent IndexError in Hungarian stemmer on empty string (PR
#3785)
- Isolate MTE tag filters per view (PR #3792)
- Fix newline in ellipsis tokens in TweetTokenizer (PR #3819)
- Isolate default WordNet reference relations (PR #3872)
- Fix resolution paths that cancel distinct literals (PR #3878)
- Fix UTF-8 text preprocessing in NKJP corpus reader (PR #3879)
- Remove shadowed FeatStruct._walk stub and preserve its docstring
(PR #3893)
- Count all increasing pairs in RIBES kendall_tau (PR #3900)
- Fix the TweetTokenizer ZWJ emoji pattern capturing adjacent
whitespace (PR #3913)
- Surface the CoreNLP server's reason for a non-2xx answer (PR
#3942)
- Omit unaligned (NULL target) pairs from Alignment strings, so IBM
model output prints (PR #3967)
- Fix DependencyGraph cycle detection, which returned False on
every parsed graph (PR #3978)
- Show, range and draw an alignment with an unaligned pair on every
surface (PR #3987)
What's Changed:
- Show README.md as the PyPI long description
- Add stopwords.words("english-extended")
- Fix pathsec.urlopen treating NO_PROXY as a configured proxy
(#3748)
- Krippendorff's alpha: treat placeholder labels as missing data
(#2865, #2732)
- Harden core security modules: pathsec URL-scheme bypass,
picklesec denylist, xmlsec parse()
- Add a living per-advisory security-probe harness
- Harden tokenize file I/O against path traversal (punkt,
stanford_segmenter)
- Add a shared pathsec sandbox conftest fixture
- Contain CorpusReader / agreement / chat80 file I/O in the pathsec
sandbox
- Write parse demo/profile output to a private staging dir, not
/tmp or the CWD
- Add pathsec regression tests for sem.util / xmlsec / dataset
loaders
- Harden maxent / named-entity model-artifact saves against
shared-tmp planting
- Route weka ARFF_Formatter.write through the pathsec sandbox
- Harden untrusted-model pickle allowlist (exact globals +
numpy.lib denylist)
- Split nltk_data pickle-asset regression into always-on and
skippable tests
- Reject Java launcher switches and @argfiles in the java() cmd
argument (CWE-88)
- Prevent infinite loop from cyclic collection indices
- Fix ARFF label injection in Weka classifier
- Stabilize multiprocessing test timeouts and context handling
- Fix Java wrappers decoding str output from java()
(universal_newlines)
- Fix shared defaults in chat80 Concept
- Harden internals.java() (CWE-88): options, classpath, env,
discovery; route MaltParser through it
- Bound decompression entry points in nltk (zip/gzip layers +
pathsec.ZipFile + weka), CWE-409
- Restore pre-commit formatting and register the new probe id on
develop
- Harden pickle deserialization (picklesec) against
arbitrary-code-execution gadgets
- Fix internals.doctest to match the hardened
_validate_java_options allowlist
- Bound re_show's caller-supplied regex against ReDoS (CWE-1333)
- Refuse CWD-relative matches for bare names in find_file/find_dir
(CWE-427)
- Bound RegexpStemmer's caller-supplied pattern against ReDoS
- Deserialize chat80 shelve values through the restricted unpickler
(RCE fix)
- Harden path/resource/tool validation in nltk.pathsec
- Harden downloader Package attributes and extraction against
traversal/Zip-Slip
- Harden nltk.xmlsec.parse: read filename sources through
pathsec.open
- Validate the NE chunker save_params destination through the
tool-directory guard
- Pin maxent scratch files to a data root and validate scratch IO
- Add staging_tempdir and tighten data.py zip/staging guards
- Stage boxer scratch input inside an nltk.data root
- Canonicalize obfuscated numeric IPv4 in the SSRF guard (CWE-918)
- Harden the VERSION read in nltk package init with
pathsec.open_package_resource
- Harden stanford tokenizer/segmenter wrappers: validate jar/model
paths and argv
- Harden tagger model loads and tool paths
(perceptron/crf/hunpos/stanford)
- Stage weka scratch files in a data root and validate the WEKAHOME
jar path
- Route corpus-reader scratch tempfiles through a data root and
label framenet path refusals
- Harden malt and stanford parser wrappers: validate
binary/model/IO paths and argv
- Harden nltk.tbl.demo model load/save through picklesec and
pathsec
- Add tests covering already-merged pathsec/security guards
- Fix develop CI: register data roots for benign save/IO tests
broken by pathsec hardening
- Prevent combinatorial DoS in skipgrams
- Prevent IndexError in Hungarian stemmer on empty string
- Sanitize SFM markers in ToolboxData
- Isolate MTE tag filters per view
- Prevent classpath shadowing in GenericStanfordParser
- Prevent batch-boundary injection in StanfordParser
- Add remaining GHSA-8mgp security regression tests
(pathsec/java/redos/archive/corpus)
- Add redos.source_of/reharden and picklesec.harden_object_graph
- Load nltk.tbl.demo models through an allowlisting unpickler
- Bound algorithmic-complexity (time) DoS in stemmers, metrics,
parsers (CWE-407 / CWE-770)
- Bound regex compilation against a compile-time DoS
- Route top-level nltk regexes through nltk.redos
(CWE-1333/CWE-400)
- Route nltk.sentiment regexes through nltk.redos
(CWE-1333/CWE-400)
- Route nltk.chat regexes through nltk.redos (CWE-1333/CWE-400)
- Route nltk.corpus regexes through nltk.redos (CWE-1333/CWE-400)
- Route nltk.parse regexes through nltk.redos (CWE-1333/CWE-400)
- Route nltk.draw regexes through nltk.redos (CWE-1333/CWE-400)
- Route nltk.stem regexes through nltk.redos (CWE-1333/CWE-400)
- Route nltk.translate regexes through nltk.redos
(CWE-1333/CWE-400)
- Route nltk.sem regexes through nltk.redos (CWE-1333/CWE-400)
- Route nltk.tree regexes through nltk.redos (CWE-1333/CWE-400)
- Route nltk.tokenize regexes through nltk.redos (CWE-1333/CWE-400)
- Route nltk.chunk regexes through nltk.redos (CWE-1333/CWE-400)
- Route nltk.classify regexes through nltk.redos (CWE-1333/CWE-400)
- Route nltk.tag regexes through nltk.redos (CWE-1333/CWE-400)
- Route nltk.app regexes through nltk.redos (CWE-1333/CWE-400)
- Route nltk.ccg regexes through nltk.redos (CWE-1333/CWE-400)
- Bound compile-time DoS (ValueError) in the concordance and nemo
GUI regex handlers (CWE-1333/CWE-400)
- Harden RegexpStemmer against pre-compiled ReDoS patterns
(CWE-1333)
- Add security regression tests + harden SSRF filter (transition
tunnels, IP literals)
- Strengthen the advisory security-probe harness (guards already on
develop)
- Guard model-artifact loaders against un-sandboxed open()
- Route security-test fixtures through the pathsec I/O helpers
- Stage REPP scratch input in an nltk.data root instead of the
shared system temp
- Bound JSON deserialization against deep-nesting crash and
resource DoS
- Fix newline in ellipsis tokens in TweetTokenizer
- Refuse NUL bytes in pathsec.validate_path; add a pathsec
attack-candidate harness
- Re-check the normalized form of a no-protocol resource path
against traversal (CWE-22)
- Add and expand security / ReDoS attack test coverage
- Escape corpus-derived text in the WordNet browser app (CWE-79
XSS)
- Add test-only regression coverage for already-fixed CWEs
- Extend the no-unsandboxed-open guard to app/draw/twitter and
route every open through pathsec.open
- Add the all-regex-through-redos pre-commit guard and two
regression tests
- Exclude pytest's ephemeral basetemp from the pickle-asset scanner
- Open operator-chosen app/twitter paths with an annotated builtin
open, not pathsec.open
- Bound the WekaClassifier model path to the pathsec data roots
- Tidy transitionparser security comments to one point per <=3
lines
- Deflake anti-DoS timing tests and harden senseval _fixXML
leading-run ReDoS
- Harden and standardize NLTK's external-tool wrappers on a pathsec
trusted-spawn API (CWE-426/427/732/93/88)
- Don't miscount a python-crfsuite Windows failure as a pathsec
over-block (tag sweep)
- Give every pathsec refusal the standard Security Violation marker
(spawn_trusted + ZIP audit)
- Bound recursion depth in parsers and traversal helpers
- Isolate default WordNet reference relations
- Harden and complete the CoreNLP server-option allowlist
(CWE-88/22/502)
- Optimize CI test matrix
- Harden CoreNLPServer start attempts
- Make DoS/ReDoS timing regression tests load-invariant (fix CI
flakiness)
- Extend no-unsandboxed-open guard to
gzip/bz2/lzma/zipfile/tempfile (+ coverage test)
- Count all increasing pairs in RIBES kendall_tau
- Potential fix for code scanning alert no. 28: Clear-text logging
of sensitive information
- Remove shadowed FeatStruct._walk stub and preserve its docstring
- Fix resolution paths that cancel distinct literals
- Add GHSA-j456 live probe; expand the GHSA-8mgp probe attack set
- Handle the redos MAX_PATTERN_LENGTH refusal in
help._format_tagset
- Handle the redos MAX_PATTERN_LENGTH refusal in
TokenSearcher.findall
- Handle the redos MAX_PATTERN_LENGTH refusal for a tgrep /regex/
node literal
- Make help._format_tagset fail closed on an oversized tag pattern
(fix develop CI)
- Add nltk.termsec terminal-output sanitizer with attack-matrix
tests
- [WIP] Fix quadratic DoS in Tree.chomsky_normal_form right-factor
binarization (GHSA-r53h-rw34-8h97)
- Bound SyllableTokenizer fallback-vowel accumulation (CWE-400 /
CWE-407)
- Fix develop CI on Python 3.10/3.11: let deepcopy reach
MAX_TREE_DEPTH
- Fix the XMLCorpusView ReDoS-test Windows CI flake (in-process op
timing + Windows autoload)
- Route every library print through the nltk.termsec.safe_print
chokepoint
- Log the perceptron training progress on the module logger (fixes
the develop 3.10/3.11 cells)
- Resolve every executed tool binary absolute-only: prover9/mace4,
megam, tadm, java, hunpos (untrusted search path, CWE-426/427)
- Pin the GHSA-cc5r-64rf-75hg proof of concept end to end and run
its sink as probe phase 6 (follow-up to #3887)
- Validate chat80.sql_query dbname for path containment (CWE-59/73)
- Extend the tool model-file guards to crf, stanford tagger and
segmenter
- Consolidate the internals, downloader, pathsec and decorators
guards from #3753 and #3850
- Consolidate the nltk/parse guards from #3753 and #3850 and judge
their inputs on real characters
- Sanitise the CSV cells and report lines the sentiment helpers
write
- Read a long line in linear time in
SeekableUnicodeStreamReader.readline (GHSA-j8g8)
- Escape AlignedSent DOT labels, refuse NUL, and keep the alignment
checks under -O
- Align quote tokens in linear time in the Treebank and NLTK word
tokenizers
- Never leave the pop of an escaped bidi override live in
sanitize_terminal
- Find readline's line boundary with str.splitlines, not a timed
regex
- Walk semi-relations once in relextract, and keep a pattern's
flags in redos.reharden
- Fix UTF-8 text preprocessing in NKJP corpus reader
- Make the CoNLL SRL, TIMIT and XML view readers linear, and parse
corpus XML through xmlsec (GHSA-v8f3)
- Measure probe scaling in process CPU time so a loaded runner
cannot flip a verdict
- ci(deps): bump dorny/paths-filter from 3 to 4 in the
github-actions group[bot] in
https://github.com/nltk/nltk/pull/3943
- Surface the CoreNLP server's reason for a non-2xx answer and warm
the live test server
- Guard the Stanford tagger and segmenter input against line
injection, verified on the real tools
- Bring the self-contained tests of #3753 to develop
- Guard the twitter CSV exports against formula and
control-sequence injection, verified on the real corpus
- Guard the maxent parameter files against outside-root paths and
unrepresentable names, verified on a real trained model
- Parse CCG lexicon categories in linear time and refuse oversized
ones, verified under attack and on the documented grammars
(GHSA-89p3-fcch-88ph, split from #3753)
- [Do not merge yet] Guard tool wrappers, Prover9 input and the
downloader against injection and unbounded-response DoS
- Added RTL language syntactic tree visualization support
- Make the word, blankline and TextTiling tokenizers, the valuation
parser, relextract and toktok linear on retried leading regex
runs
- Add the 3.10.4 commit authors and security reporters to
AUTHORS.md
- Pair the scaling samples by rep and size the under-margined
probes over the floor
- Trusted spawn: O_NOFOLLOW and inode check on the final file, link
owner and control-character checks, refusal reasons, argv[0] as
invoked; real Graphviz in CI (#3887 follow-up)
- Support symlinked executables in pathsec
- [Do not merge yet] Keep downloaded archives private and serve
other accounts through extraction (fixes#3928)
- Bound algorithmic-DoS and uncontrolled-recursion vectors
(CWE-400/407/674/1333)
- Guard the two-class any-character runs and count apostrophe-named
groups (follow-up to #3895)
- Fix#3394: Ensure punkt_tab is downloaded when punkt is requested
- Restore add_blank_lines and to_settings_string after the
GHSA-f2h2 depth bound
- Let chat80.val_load read back what val_dump wrote
- Keep full collections out of timed scaling samples (89p3 teeth
and relextract reds)
- CI: take the Windows Graphviz from the cached third-party tools,
not Chocolatey
- Close the chat80 store swap window and read stores by a key
manifest
- Route every module's file removal, rename, privacy test and
sqlite open through pathsec
- Fix the develop CI breakage from the fg9w advisory merge (Senna
-path separator, test fixtures under /tmp)
- Refuse relation walks deeper than MAX_RECURSION_DEPTH instead of
truncating them
- Harden the Toolbox reader and writers after the #3958 re-audit
- Route the downloader, XML corpus view, internals and Toolbox
through nltk.xmlsec and nltk.data
- Open gzip file names through pathsec, and refuse a linked PanLex
Lite database
- Fix DependencyGraph cycle detection, dead on every parsed graph
- Bound YAML deserialization against depth and resource DoS,
verified on the real loaders
- Guard java() cmd[0] against JEP 330 source-file execution,
verified on a real JDK
- Make Tree.__repr__ iterative so a deep accepted tree does not
overflow the stack
- Create perceptron model files exclusively under the pinned
directory (issue 3980)
- Drop the dash-banner section dividers from the model-artifact
tests
- Skip the real-launcher test when java resolves to a runtime-less
stub
- Pin the CI megam download to https and its sha256
- Run the advisory probes only on a full-matrix run; keep the unit
tests out of the wheel
- Charge the shipped-form teeth leg on CPU time (Windows regex
timeout is wall-clock)
- Prepare NLTK release 3.10.4
- Rewrite the 3.10.4 release notes around the user-facing security
changes
- Omit NULL targets from alignment strings
- Fix TweetTokenizer ZWJ pattern capturing whitespace
- Show, range and draw an alignment with an unaligned pair on every
surface (follow-up to #3967, #1885)
- Pin the TweetTokenizer ZWJ alternative: linear on repeated
shapes, no whitespace absorbed (follow-up to #3913)
- Credit every contributor since 3.10.3 in the 3.10.4 notes and
AUTHORS.md; name the PRs merged after #3986
- Make the span_tokenize teeth control compute-bound so Windows
reads its quadratic (deflake, test-only)
Signed-off-by: Leon Anavi <leon.anavi@konsulko.com>
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
Upgrade to release 9.1.0:
- Removed support for Python 3.10, now end-of-life.
- Where a path supplies more than one metadata directory for a
distribution, as an interrupted or unclean upgrade may leave
behind, discovery now prefers the greater one by reverse
lexicographic ordering of the directory names, so the stale
metadata no longer shadows the version in use.
- A missing required metadata field (Name or Version) now raises a
KeyError that names the field and identifies the distribution,
which PathDistribution renders as its metadata path.
- Restored lazy validation of EntryPoint.value, deferring the
ValueError for an invalid value until a component is accessed or
the entry point is loaded, so one malformed entry point no longer
prevents entry points in other groups from being selected.
- importlib_metadata.diagnose no longer crashes on a metadata
directory that has no metadata file or whose metadata declares no
name. Such distributions are now reported by their location.
Signed-off-by: Leon Anavi <leon.anavi@konsulko.com>
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
Upgrade to release 2.5.0:
- imgtool now requires Python 3.8 or later. This also fixes key
export commands, which failed on Python 3.8 and 3.9.
- Added imgtool keyinfo, which reports whether a PEM file holds a
key pair or only a public key. With --require, it fails if the
kind does not match, which build systems can use as a check.
- Added --name-suffix to imgtool getpub and imgtool getpubhash,
which appends a suffix to the generated C or Rust symbol names so
that several keys can be embedded in one image.
- Public-key-only PEM files are now supported by imgtool getpub,
imgtool getpubhash and imgtool verify. imgtool sign given a
public-only PEM now fails with a clear error.
Signed-off-by: Leon Anavi <leon.anavi@konsulko.com>
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
Upgrade to release 2.16.0:
Changed:
- Look up only the cookie domains that match the request host, and
skip the expired cookie sweep until a cookie is due. With 1,000
cookies across 500 domains, building the Cookie header is about
18 times faster. Cookies from different domains are now sent from
the least to the most specific domain.
Fixed:
- Reuse the client cookie jar instead of copying it on every
request and redirect. A custom policy set with
client.cookies.jar.set_policy() now applies to outgoing requests
Signed-off-by: Leon Anavi <leon.anavi@konsulko.com>
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
Upgrade to release 7.17.0:
- Dropped support for Python 3.10.
- Binary wheels are now provided for iOS and Android. This allows
the use of the optimized C tracer on mobile platforms.
- Fix: three or more levels of nested with statements could make a
with line look like a branch with an exit that can never happen.
Using the sys.monitoring core, these phantom branches were
reported as partial. Other cores hid the problem but still
counted the phantom branches in the total.
- Fix: a comment or string that merely mentioned if True: or while
True: could mark its line as a branch that is known to be
partial, so a branch that never ran was reported as taken. These
built-in patterns now only match at the start of the statement.
- Fix: region detection missed functions defined in unusual places,
such as else and except clauses. This is now fixed.
- Fix: recent versions of the lcov tool complain if the same
function name appears more than once in a report. Python can have
multiple same-named functions in a few ways, for example,
getters/setters or conditionally defined functions. These are now
disambiguated by adding a @<line number> suffix to the function
name.
- Fix: The "Can't combine statement coverage data with branch data"
error now names the two data files it couldn’t combine. The error
description in the docs now lists common causes and how to find
the process that wrote the file.
- Fix: at coverage totals close to 0 or 100, the XML report could
get incorrect line-rate values like 8.333e-05 or 1. This is now
fixed.
- Fix: a parallel data file saved more than once kept the hash of
its first save in its file name. Combining could then skip it as
a duplicate of another file that had the same data at that point,
and delete it, losing the data recorded after the first save.
Each save now updates the hash. This had been a problem since
7.13.1.
- Fixes for obscure but legal file names:
- File names (and plugin names) can contain control characters,
which are legal in POSIX paths but unsafe in reports: in XML
1.0 they make the document non-well-formed, and in a terminal
they act as escape sequences (colors, title/clipboard
sequences, cursor moves that forge output). Now the XML report,
the text and Markdown summary reports, and coverage debug data
replace non-printable characters in names. Printable names are
unchanged.
- A file pattern (from include, omit, or a [paths] alias)
containing a newline sent the glob-to-regex translation into an
infinite loop, hanging the process. Newlines in patterns are
now handled like any other character.
- Fixes in the implementation of the C tracer:
- If a plugin’s dynamic_source_filename method raised an
exception, lines in that function could be recorded as
belonging to a different file. Also fixed some minor memory
leaks in the C tracer.
- The refcount of None was mismanaged during function calls,
including comprehensions. This was unlikely to cause an actual
problem, but is now fixed. Thanks, Andrew Dalke.
- The C tracer held borrowed references to dictionary values,
which was unsafe on free-threaded Python. Now it takes its own
references.
- An internal error while recording a new file could be reported
as a confusing SystemError instead of the actual exception.
- Performance: the text summary report (coverage report) held on to
every file’s parsed source until it finished, so its memory use
grew with the size of the project: ten times or more what 7.14
used on large projects. Now it keeps only what it needs to print.
- Performance: HTML reports should be about 5% faster to produce.
- Kept the political banner in the README.
Signed-off-by: Leon Anavi <leon.anavi@konsulko.com>
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
Upgrade to release 26.10.2:
- Fix reading the TLS peer certificate
(TransportDetails.peer_cert) with pyOpenSSL 26.2.0 and later:
autobahn.twisted.util.extract_peer_certificate() raised
AttributeError: 'X509' object has no attribute 'get_extension'
for every certificate with extensions, as pyOpenSSL removed its
X.509 extension API (and deprecates the rest of its X509 API).
The certificate is now read with cryptography, in the new
backend-neutral autobahn.util.peer_certificate_details(), which
also never raises on certificate content it cannot parse (it is
peer data, read before authentication): extensions, or a subject
or issuer, that cannot be read are left empty. The peer_cert dict
keeps its keys and formats, except: ⚠ extensions[].value was
OpenSSL's text; a subject or issuer alternative name is still
given as OpenSSL prints it (DNS:localhost, IP Address:127.0.0.1),
other extensions now as cryptography prints them; and
signature_algorithm is ed25519 instead of ED25519 for Ed25519
certificates
- Fix the TLS peer certificate never being available on Twisted
WebSocket and RawSocket connections: it was read when the TCP
connection was made, before the TLS handshake, and so was always
None. It is now read when the TLS handshake is complete, together
with the TLS channel ID. With this, TLS client certificate
authentication (Crossbar.io authmethod tls, and dynamic
authenticators reading transport.peer_cert) works again
- Fix asyncio: fill TransportDetails.peer_cert (it was always
None), and treat a TLS server connection without a client
certificate as secure, with a TLS channel ID: is_secure was
derived from the peer certificate, so WAMP-cryptosign with
channel binding failed on asyncio servers
- Behaviour change - the TLS channel ID tls-unique on Twisted now
follows RFC 5929 (the first TLS Finished message of the
handshake, in a full handshake the client's), as Python's ssl
module, and so autobahn on asyncio, already did. Twisted used the
server's Finished message, so WAMP-cryptosign with channel
binding failed between Twisted and asyncio peers over WebSocket,
and between Twisted RawSocket peers (whose protocols did not set
is_server, so the client side used the wrong message). Twisted
peers on autobahn 26.10.1 or older and on 26.10.2 no longer agree
on the channel ID over WebSocket: upgrade both sides of
connections that use channel binding (#2014) Work around a PyPy
bug: PyPy's ssl module (up to at least PyPy 7.3.23) returns the
server's TLS Finished message from
get_channel_binding("tls-unique") instead of the first (the
client's), so autobahn on asyncio on PyPy disagreed with all
other peers on the channel ID. The new autobahn.util.tls_unique()
uses PyPy's own tls_unique_cb() on PyPy, which returns the right
message
- TransportDetails.is_server is now set on Twisted RawSocket and
asyncio WebSocket connections (it was None)
Signed-off-by: Leon Anavi <leon.anavi@konsulko.com>
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
Upgrade to release 1.20.1:
bug:
- [bug] [autogenerate] Fixed issue where a table or column comment
set to an empty string would cause autogenerate to detect a
comment change on every run. As most backends, including
PostgreSQL, MySQL, MariaDB and Oracle, don't store an empty
string comment distinctly from no comment, autogenerate now
treats an empty string comment the same as None when comparing
comments. Additionally, an empty string comment is now rendered
in op.create_table() and op.add_column() directives, where it was
previously omitted. Pull request courtesy vineoy.
- [bug] [autogenerate] [postgresql] Fixed rendering of the
PostgreSQL ~sqlalchemy.dialects.postgresql.DOMAIN type in
autogenerated migration scripts. Previously the type was
rendered using repr(), which omitted the check, not_null,
default, collation, constraint_name, schema and create_type
arguments and did not apply a dialect prefix to the underlying
data type. The full set of arguments is now rendered and the
underlying data type is rendered using the standard
type-rendering logic. Pull request courtesy Sebastian Cao.
- [bug] [autogenerate] [postgresql] Fixed false-positive "Computed
default cannot be modified" warning emitted during autogenerate
when comparing PostgreSQL Computed columns whose reflected SQL
expression includes type cast specifiers such as ::regconfig or
::text, which PostgreSQL adds to the expression as stored. These
cast specifiers are now stripped before comparison so that
otherwise equivalent expressions no longer trigger the warning.
Pull request courtesy Sebastian Cao.
- [bug] [autogenerate] [postgresql] Fixed issue where autogenerate
would detect an unchanged PostgreSQL expression-based index as
modified when the expression included type casts, such as a cast
to a type with a length or precision, a nested cast, or a cast
followed by a DESC modifier.
- [bug] [postgresql] Fixed bug where Operations.alter_column()
against a PostgreSQL identity column would render an incomplete,
invalid ALTER TABLE ... ALTER COLUMN statement with no clause at
all when existing_server_default was omitted, even if there were
no existing server default. Pull request courtesy Riti Grover.
- [bug] [environment] Fixed issue where a third party plugin
published using the alembic.plugins entrypoint, configured to
refer to a plugin module as documented at
alembic.plugins.toplevel, would cause import alembic to fail with
a TypeError, as the loaded module was erroneously iterated rather
than being set up directly. An entrypoint that refers to a
collection of modules, the form which inadvertently worked
previously, continues to be accepted but now emits a deprecation
warning.
- [bug] [autogenerate] Fixed issue where dialect-specific keyword
arguments on a ForeignKeyConstraint, ForeignKey or
CheckConstraint, such as postgresql_not_valid, were not rendered
for the constraints that are part of an autogenerated
Operations.create_table() directive, nor for an autogenerated
Operations.create_check_constraint() directive. The
Operations.create_table() case additionally requires SQLAlchemy
2.0.55 / 2.1.5 or greater.
- [bug] [autogenerate] Fixed issue where, when using SQLAlchemy
2.1, the CHECK constraint generated by a type such as
~sqlalchemy.types.Enum or ~sqlalchemy.types.Boolean with
create_constraint=True would be rendered by autogenerate as
explicit sa.CheckConstraint() entries within op.create_table(),
in addition to the type itself which already produces that
constraint, leading to a duplicate constraint error when the
migration is run.
- [bug] [operations] Fixed issue where a CreateTableOp produced
from an existing ~sqlalchemy.schema.Table using
CreateTableOp.from_table() would emit the CHECK constraint
generated by a type such as ~sqlalchemy.types.Enum or
~sqlalchemy.types.Boolean with create_constraint=True twice in
the resulting CREATE TABLE statement when invoked directly.
Signed-off-by: Leon Anavi <leon.anavi@konsulko.com>
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
Drop the Cython version patch, as upstream has a wider range now.
async-timeout was removed as an dependency upstream in v1.90.0.
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
Upgrade to addresses eMudhra certificates issue.
See https://emudhra.we-have-a-problem.com/
Sync all grpc recipes to the same version.
Removed obsolete patches:
* The compiler-argument fix is already handled upstream using
shlex.split()
* Bundled protobuf now restricts musttail to AArch64 and x86-64, so the
MIPS exclusion patch is unnecessary.
License-Update: switched from license to license expression
orig: License: Apache License 2.0
New: License-Expression: Apache-2.0
Signed-off-by: Peter Marko <peter.marko@siemens.com>
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
Upgrade to addresses eMudhra certificates issue.
See https://emudhra.we-have-a-problem.com/
$ git log --oneline v1.84.0..v1.84.1
b0c0c4f882 (HEAD -> v1.84.x, tag: v1.84.1) [Release] Bump version to 1.84.1 (on v1.84.x branch) (#43709)
405e6a1c30 [Backport][v1.84.x][security] Update gRPC repo roots. (#43700)
826616aed4 [Backport][v1.84.x][CI] Various macOS CI reliability fixes (#43701)
49d6985875 [PSM Interop][Backport v1.84.x] Update xDS legacy test scripts for health proto pa… (#43572)
Signed-off-by: Peter Marko <peter.marko@siemens.com>
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
The object id counter in src/weak_ref.rs is a std AtomicU64, which is
not available on targets without 64-bit atomics, so the build fails
for riscv32:
error[E0432]: unresolved import `std::sync::atomic::AtomicU64`
Add a patch that keeps the 64-bit counter in a Mutex instead. The code
has since been removed upstream.
AI-Generated: Uses Claude Code
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
do_configure reads the per architecture cross-answers file from the
layer by path rather than through SRC_URI, so editing it does not
change the task signature, and existing builds keep the old answers
until they are reconfigured by hand.
Make the answers directory a variable and add the file to
do_configure[file-checksums].
AI-Generated: Uses Claude Code
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
The riscv32 cross answers were derived from arm, but riscv32 has no
legacy 32-bit credential syscalls such as setresuid32, so samba asks
"Checking whether we can use Linux thread-specific credentials" instead
of the "with 32-bit system calls" variant. With no answer, configure
fails:
Checking whether we can use Linux thread-specific credentials : not found
...
The configuration failed
Answer it like riscv64 and aarch64 do.
riscv32 also has a 64-bit time_t, so the time_t maximum value check
succeeds like on 64-bit targets. Answer it OK so that samba does not
limit TIME_T_MAX to INT32_MAX.
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
src/lib/util.h uses bool without including stdbool.h, which fails to
build where no other header pulls it in, e.g. with clang on riscv32
musl:
src/lib/util.h:68:1: error: unknown type name 'bool'
Add a patch that includes it.
AI-Generated: Uses Claude Code
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
It depends on glycin, which is not available on riscv32, as it
requires libseccomp, which is not ported to rv32.
AI-Generated: Uses Claude Code
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
crossbeam-utils 0.8.16 does not recognize the riscv32-yoe-linux-musl
target as lacking 64-bit atomics, and rustix 0.38.20 with linux-raw-sys
0.4.10 lacks ioctl definitions for riscv32, so the build fails.
Add a patch that updates them in Cargo.lock to crossbeam-utils 0.8.23,
rustix 0.38.44 and linux-raw-sys 0.4.15, and regenerate sd-crates.inc.
AI-Generated: Uses Claude Code
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
When cross compiling, ibus builds unicode-parser with CC_FOR_BUILD and
the glib flags from PKG_CONFIG_FOR_BUILD. That defaults to the native
pkg-config binary, but with the target PKG_CONFIG_PATH and sysroot, so
the build tool is compiled against the target glib headers. On 32-bit
targets glib's static assertions then fail with the 64-bit host
compiler:
glib/gmacros.h:910:31: error: static assertion failed: "Expression evaluates to false"
Pass pkg-config-native, which uses the native sysroot, as
PKG_CONFIG_FOR_BUILD.
AI-Generated: Uses Claude Code
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
rice-proto counts agents in a core AtomicU64, which is not available on
targets without 64-bit atomics, so it fails to build for riscv32:
error[E0432]: unresolved import `core::sync::atomic::AtomicU64`
Add a patch that uses AtomicUsize for it, like the other id counters
in rice-proto, keeping the u64 agent id.
AI-Generated: Uses Claude Code
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
Add patches to the vendored crates:
- libc: Add the syscall numbers 443 to 450, which riscv32 musl lacks
and rustix needs for SYS_futex_waitv.
- fff-search, rmcp, nu-command and nu-dap: Use the 64-bit atomics from
portable-atomic on targets without 64-bit atomics, where std does not
provide them, and record the dependency in Cargo.lock since the build
uses --frozen.
AI-Generated: Uses Claude Code
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
std only provides AtomicU64 on targets with 64-bit atomics, so ruff
fails to build for riscv32:
error[E0432]: unresolved import `std::sync::atomic::AtomicU64`
--> crates/ruff/src/cache.rs:7:25
Add a patch that uses portable-atomic's AtomicU64 for the cache
timestamp on such targets, archived in the same on-disk format.
AI-Generated: Uses Claude Code
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
floret keeps its training token counter in a std::atomic<int64_t>. On
riscv32 its operations become __atomic_*_8 libcalls, and floret is not
linked with libatomic, so linking the CLI fails:
ld.lld: error: undefined symbol: __atomic_fetch_add_8
Add a patch that checks in CMake whether 64-bit atomics need libatomic
and only then links the libraries with it and adds it to floret.pc.
AI-Generated: Uses Claude Code
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
The vendored libc 0.2.171 crate declares getrandom() for musl only on
some architectures, so the vendored getrandom crate fails to build for
riscv32 musl:
error[E0425]: cannot find value `getrandom` in crate `libc`
Backport the libc 0.2.172 change that declares it for all musl targets,
and update the vendored crate checksums of the modified files.
AI-Generated: Uses Claude Code
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
klibc only has a riscv64 port, which hardcodes KLIBCBITSIZE = 64 and
uses 64-bit loads and stores in its assembly, and upstream has no
riscv32 port. With KLIBC_ARCH defaulting to TARGET_ARCH, the build fails:
make[1]: *** No rule to make target '.../usr/klibc/arch/riscv32/MCONFIG'. Stop.
Mark klibc and the recipes sharing klibc.inc incompatible with riscv32
so they are skipped instead of failing to build.
Also skip all recipes inheriting klibc
AI-Generated: Uses Claude Code
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
The encoder uses atomic operations on 64-bit counters. On riscv32 these
become __atomic_*_8 libcalls, and libx265 is not linked with libatomic,
so linking the x265 CLI fails:
ld.lld: error: undefined reference: __atomic_fetch_or_8
>>> referenced by libx265.so.217 (disallowed by --no-allow-shlib-undefined)
Add a patch that checks in CMake whether 64-bit atomics need libatomic
and adds it to the platform libraries only in that case.
AI-Generated: Uses Claude Code
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
Ricardo Salveti
Khem Raj
Ankur Tyagi
Leon Anavi
Khem Raj
Ross Burton
Peter Marko