Upgrade to release 4.3.1:
- Fix type-check failures with ty 0.0.87
- Exclude .DS_Store from release
- Route annotated command help through the application output
stream
- Add example embedding a cmd2 app in a Textual app with
textual-tty
- Add example embedding a cmd2 app in a Textual app with
textual-tty
Signed-off-by: Leon Anavi <leon.anavi@konsulko.com>
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
Upgrade to release 7.2.5:
- Fix @cachedmethod with condition for decorated methods taking a
pending keyword argument (broken since v7.0.0).
- Fix pickling and deep-copying of instances using @cachedmethod
(broken since v7.0.0).
- Fix calling @cachedmethod decorated methods through the class
with the receiver (self) passed as a keyword argument (broken
since v7.2.4).
- Various documentation improvements.
Signed-off-by: Leon Anavi <leon.anavi@konsulko.com>
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
Upgrade to release 3.13.0:
- It is now possible to nest ThreadSensitiveContext by providing an
explicit executor for thread-sensitive tasks. This enables moving
a block of such work away from the parent context's shared worker.
The provided executor must run work sequentially and on a single
thread, as per ThreadPoolExecutor(max_workers=1), and the calling
code is responsible for managing the executor's lifecycle.
- Local attribute access is now 2-3x faster. Local(...) now uses one
of two internal specialisations, selected by thread_critical,
instead of checking the mode on every access.
As a result Local is now marked @final and raises TypeError if
subclassed. Subclassing was never supported, and could not have
worked correctly with the specialised implementations.
- asgiref.timeout.timeout is deprecated, and will be removed in a
future release. Move to use asyncio.timeout() instead.
- Added support for Python 3.15.
- Dropped support for EOL Python 3.10.
Signed-off-by: Leon Anavi <leon.anavi@konsulko.com>
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
pyproject.toml specifies setuptools.build_meta as the build backend, so
inherit python_setuptools_build_meta instead of setuptools3.
Fixes:
QA Issue: inherits setuptools3 but has pyproject.toml specifying backend
setuptools.build_meta, use the correct class [pep517-backend]
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
- On the mini bar, an overlong filename gets properly clipped.
- The keypad operator keys produce their inscribed characters.
Signed-off-by: Ankur Tyagi <ankur.tyagi85@gmail.com>
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
- Recognize timezone file names with multiple slashes in the TZ environment
variable.
- Use HTTPS in place of FTP to retrieve tzdata archive.
Signed-off-by: Ankur Tyagi <ankur.tyagi85@gmail.com>
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
simple-websocket is the only hard dependency in pyproject.toml, and the
default threading async mode uses it for WebSocket connections. The
threading driver and the client also import queue (python3-threading),
and the client imports ssl (python3-io).
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
Simple WebSocket server and client for Python, built on wsproto.
Backport the upstream conversion of the async tests to pytest-asyncio;
the 1.1.0 tests call asyncio.get_event_loop(), which fails on Python
3.14 when there is no current event loop. Add it to the fast ptest list;
all 47 tests pass on qemuriscv64.
AI-Generated: Uses Claude Code
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
Since 1.14.0 upstream no longer builds the C++ library (libhdf5_cpp,
libhdf5_hl_cpp) by default, so recipes that need it fail to find it.
Add a "cpp" PACKAGECONFIG to turn on HDF5_BUILD_CPP_LIB, keeping it
disabled by default.
The C++ build also installs the h5c++ compiler wrapper, which needs the
same RECIPE_SYSROOT_NATIVE cleanup and multilib script handling as h5cc.
Fixes: https://github.com/openembedded/meta-openembedded/issues/814
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
Its added to OE-core via below commit
commit f9903f74ee36de4dd592f3bc8b1ce7b08f32608c
Author: Ross Burton <ross.burton@arm.com>
Date: Fri Oct 17 14:16:25 2025 +0100
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
The CMake build registers every test with ctest, including the ones
upstream's Makefile keeps in BIGTESTS (and Bazel marks size = "large"),
which "make test" does not run. Each of these runs silently until it
finishes, and under qemu emulation several of them take longer than the
ptest-runner timeout. On qemuriscv64, dfa_test stalled in
Multithreaded.SearchDFA until ptest-runner killed the whole suite after
450s. Under qemu-user, exhaustive3_test takes ~460s, random_test ~760s,
and dfa_test more than 15 minutes.
Exclude dfa_test, exhaustive*_test and random_test from run-ptest when
running under QEMU, detected at runtime via DMI sys_vendor, a fw-cfg
device tree node or a qemu device tree compatible. Doing this at build
time with a qemuall override would make re2's task signatures differ
between machines sharing a tune. Real hardware still runs all tests.
AI-Generated: Uses Claude Code
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
"Whois request" and "AS resolver - IPv6" only query public whois
servers and upstream notes they often fail. "AS resolver - IPv6" failed
intermittently in testimage on qemuarm64 and passed on a rerun. Tag both
with skipped_ptest so run-ptest leaves them out.
AI-Generated: Uses Claude Code
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
Changelog:
* Drop Python 2 support; require Python 3.5+
* Remove the module-level names INT_TYPES, STR_TYPES and xrange
* Add inline type annotations; IPy is now a package that ships py.typed
* Fix IP.broadcast() returning an IPv4 address for IPv6 networks
* Fix IP.reverseName() for 0.0.0.0/0
* Fix the flaky IPSet fuzz test
The fuzz test fix stops the intermittent ptest failure in
TestIPSet.testRandomContains, whose oracle ignored that IPSet merges
adjacent prefixes into a larger one.
The sdist is now named ipy-1.2.tar.gz, so drop PYPI_PACKAGE_SDIST.
AI-Generated: Uses Claude Code
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
This release is mostly hardening against untrusted repository content.
It fixes two checkout issues: a crafted whiteout name that strips to
".." and makes a --whiteouts checkout delete the destination's parent
(GHSA-cp2j-qv7x-wcw4), and union checkouts following a symlink planted
at a directory name and writing outside the destination
(GHSA-7pq7-3pmj-9wp6). Composefs and commit signature checks with the
OpenSSL ed25519/SPKI backends now accept only EVP_DigestVerify() == 1,
where error returns used to count as a valid signature. Static delta,
content header, commit and curl header parsing gain bounds checks.
Behaviour changes: the default min-free-space is now min(3%, 1GB), and
the systemd generator leaves boot.mount to systemd-fstab-generator when
/etc/fstab has a /boot entry.
https://github.com/ostreedev/ostree/releases/tag/v2026.5
AI-Generated: Uses Claude Code (Claude Opus 5.5)
Signed-off-by: Ricardo Salveti <ricardo.salveti@oss.qualcomm.com>
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
The test expects a message sent from another thread 0.2s later to arrive
within a 0.5s poll window, which a loaded test host misses. Upstream
marks it flaky(reruns=3), but pytest-rerunfailures is not available.
Adjust the musl sed to the new end of the -k expression.
AI-Generated: Uses Claude Code
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
test_async_cat_file_ranges fetches a file from a public S3 bucket, which
fails without DNS and internet access on the target, as in sysvinit
qemu images where resolv.conf is empty.
AI-Generated: Uses Claude Code
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
run-ptest wrote each test's output into a file in the image, so an
intermittent gpiod-glib-test failure in testimage left nothing to debug.
Print the output when a test fails.
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
mount/blkdev-fsname-hello{,-ll} attach a loop device, but loop is built
as a module and was not in the ptest image, so losetup failed. Recommend
kernel-module-loop and modprobe it in run-ptest, since it is not loaded
on demand when /dev/loop-control has not been created.
AI-Generated: Uses Claude Code
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
musl defines CLOCK_SGI_CYCLE, so on aarch64 cycle.h uses
clock_gettime(CLOCK_SGI_CYCLE) as its tick counter. Linux returns
EINVAL for that clock, every measurement reads as zero, and
FFTW_MEASURE planning loops forever, which times out the bench ptest.
Skip the SGI clock on Linux.
AI-Generated: Uses Claude Code
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
Upgrade to release 10.17.0:
- Added {class}pikepdf.ObjectSet, which records the objects that a
walk of a PDF's object graph has already visited. Objects in a
PDF are shared and may refer back to their ancestors, so a search
that only limits its depth can take exponential time on quite
ordinary files. ObjectSet.add() reports whether an object is new,
and tells apart direct objects and objects of different Pdf,
which a set of objgen cannot. See {ref}walking-object-graphs.
- Added {meth}pikepdf.Object.is_same_object_as, which tests whether
two references lead to the same object. Neither is, == nor
comparing objgen answers this reliably.
- Added {meth}pikepdf.Pdf.content_scopes and
{meth}pikepdf.Page.content_scopes, which iterate over every
content stream that pages can draw, each with the resource
dictionary it uses: the page itself, form XObjects, annotation
appearance streams, tiling patterns, Type 3 glyph procedures and
soft masks, however they are nested or shared. This is the way to
find every font, image or colour space a document uses. Each
{class}pikepdf.ContentScope is reported once for the document;
pages= selects pages and kinds= selects what is reported.
- Added {meth}pikepdf.Page.get_resources, which returns the
resources of a page or form XObject for reading. Unlike
{attr}pikepdf.Page.resources, it finds resources inherited from
the page tree, does not create an empty dictionary when there are
none, and returns None instead of raising when /Resources is
malformed.
- Fixed comparing two objects with == taking exponential time when
the objects being compared share children, such as the same file
opened twice. Each pair of objects is now compared once.
- Fixed {meth}pikepdf.Pdf.add_pages_from hanging on a page with an
action whose /Next actions are shared or form a loop.
- Fixed {meth}pikepdf.Pdf.open_outline reading only the first item
at each level of an outline whose items are direct objects. Such
items are now made indirect when the outline is saved, as the PDF
specification requires.
- Reading a stream that will not decode now raises
{exc}~pikepdf.exceptions.DataDecodingError with the reason,
whether the stream's data was read from a file or set from
memory. Previously a stream read from a file raised a
{exc}~pikepdf.exceptions.PdfError that said only "read_bytes
called on unfilterable stream", and the reason was available only
from {meth}pikepdf.Pdf.get_warnings. DataDecodingError is a
PdfError, so existing handlers are unaffected.
- An exception raised by a JBIG2 decoder now reaches the caller of
{meth}pikepdf.PdfImage.as_pil_image,
{meth}pikepdf.PdfImage.extract_to and
{meth}pikepdf.PdfImage.read_bytes unchanged, with its own type,
message and traceback. Previously, for an image read from a file,
it was replaced by a generic DataDecodingError or PdfError.
- Added {exc}~pikepdf.exceptions.QpdfRuntimeError, raised for
errors from qpdf that pikepdf has no more specific exception for.
These were previously raised as a bare RuntimeError, which except
pikepdf.PikepdfError did not catch. The new exception derives
from both {exc}~pikepdf.exceptions.PikepdfError and RuntimeError,
so handlers written for RuntimeError still work.
- The empty name / is now supported. The PDF 1.7 and 2.0
specifications define a lone slash as a valid name made of zero
characters, but pikepdf.Name('/') raised ValueError, and
dictionaries refused / as a key. This broke
{meth}pikepdf.Pdf.add_pages_from on files with an empty-named
destination such as /Dests << / 3 0 R >>. Such names can now be
created, used as dictionary keys, and migrated during page
copying. bool(Name('/')) is False, like an empty
{class}pikepdf.String. Name(''), which lacks the leading slash,
is still rejected. PDF/A does not forbid empty names, and the
PDF/A validator accepts them.
- AcroFormField.get_inheritable_field_value_as_name() now returns a
{class}pikepdf.Name, as documented, instead of a str. When the
value is missing or not a name, it returns the empty name
Name('/'), which is falsy, instead of ''. Comparisons such as ==
'/Tx' are unaffected.
- Fixed an out-of-bounds read when opening a PDF from a Python file
object whose readinto() reports more bytes than were requested.
This now raises ValueError, as writing to such a stream already
did. Non-blocking streams, whose readinto() returns None, now
raise ValueError instead of hanging.
- Fixed building from source on PyPy, which failed to compile since
v10.0. PyPy now reads Python file objects through readinto(),
like CPython, instead of copying the result of read().
- {func}pikepdf.open reads directly from the file descriptor when
given a filename, instead of calling the seek() and readinto()
methods of a Python file object for every read. Opening and
reading a large file with many objects is about four to five
times faster, matching the speed of memory mapping
(access_mode=AccessMode.mmap). Streams passed to
{func}pikepdf.open, including plain file objects, are read
through their methods as before.
- Documented {class}pikepdf.TokenFilter with worked examples that
shorten numbers and convert RGB colors to grayscale, and
explained when to use a token filter instead of
{func}pikepdf.parse_content_stream. The API reference no longer
says that the parser should not be used to edit content streams.
The TokenFilter.handle_token documentation now says correctly
that exceptions it raises propagate to the caller.
Extend RDEPENDS with the missing runtime dependencies
python3-fonttools and python3-jsonschema which are required
to import pikepdf.pdfa.
Install pdfa_differential.py to pass the tests from
test_pdfa_differential.py. Fixes:
E ModuleNotFoundError: No module named 'pdfa_differential'
This work was sponsored by GOVCERT.LU.
Signed-off-by: Leon Anavi <leon.anavi@konsulko.com>
Signed-off-by: Khem Raj <raj.khem@gmail.com>
Upgrade to release 3.10.4:
This release routes every file path, archive, download, URL fetch and
external program through one sandbox that keeps reads and writes inside
the trusted data directories, refuses path traversal, symlink escapes
and decompression bombs, blocks requests to internal network addresses,
and runs Java and other third-party tools only from trusted absolute
locations with validated options and input. Pickled models now load
through a restricted unpickler, XML and JSON parsing are hardened, and
regular expressions, recursion and slow algorithms in the tokenizers,
parsers, stemmers and corpus readers are bounded against
denial-of-service input. Text that NLTK writes to the terminal, to CSV
files, to the WordNet browser and to third-party tool input is sanitized
against injection.
Security changes you may notice
Every refusal below raises with a message that starts with "Security
Violation" and names the check, so a failure is easy to search for (PR
- Data directories (nltk.pathsec, nltk.data): corpora, models and
every other resource are read and written only inside the
directories listed in nltk.data.path. A path that leaves a data
root, a symlink that points out of one, and a NUL byte or encoded
traversal in a resource name are refused. A data root, or a
directory on the way to a resource, that another account can
write to is skipped with a warning naming the directory and the
remedy. Keep private data in a private directory and add it to
nltk.data.path; /tmp is not a data root. Model save and load
methods (perceptron, maxent, named-entity chunker, Weka, crf,
hunpos, Stanford) accept destinations inside a data root only (PR
#3797, #3753, #3850, #3761, #3755, #3759, #3760, #3763, #3806,
#3807, #3808, #3809, #3813, #3814, #3815, #3848, #3851, #3852,
#3853, #3863, #3871, #3888, #3945, #3961, #3969, #3981)
- Downloader: downloaded package archives stay private to the
installing account (0600). An install another account must read
is extracted: a download made as root or outside the installing
account's home extracts on its own, and nltk.download(...,
extract=True) / python -m nltk.downloader --extract (or
--no-extract) decide explicitly. This also unpacks the packages
the index leaves zipped (wordnet, omw-1.4 ...); the extracted
files are readable by every account and are found before the
archive. find() names an archive it cannot read and says how to
extract it instead of reporting it "not found". The downloader
installs only into directories no other account can write to,
treats an archive another account owns or can write to as stale
and downloads it afresh, extracts only from an archive no other
account could have changed since its checksum was verified,
bounds the size of what it fetches, and refuses index entries and
archive members that would write outside the package directory
(PR #3929, #3804, #3850, #3778)
- Pickles: nltk.data.load(..., format="pickle") reconstructs plain
data only; a pickle that names any class or function is refused.
Model loaders that need classes use an explicit allowlist of
audited globals, so a gadget such as os.system can never be
reconstructed from a model file. chat80 shelve values load the
same way (PR #3794, #3754, #3757, #3802, #3817, #3823, #3824)
- XML, JSON and YAML: nltk.xmlsec refuses any document that
declares an entity or references an external one (billion-laughs,
external-entity and DTD attacks) and bounds the tree depth, and
every corpus XML view, the downloader index, Toolbox and
internals parse through it. JSON resources load with a depth and
size bound; YAML (the markdown corpus front matter and
nltk.data.load(..., format="yaml")) loads through a bounded safe
loader. Oversized or too deeply nested input raises ValueError
naming the limit (PR #3754, #3805, #3828, #3933, #3972, #3974)
- Regular expressions (nltk.redos): every regular expression NLTK
compiles, including one you pass to RegexpTokenizer,
RegexpStemmer, RegexpParser, re_show, TokenSearcher.findall,
tgrep, nltk.help tag-set lookups and the concordance and nemo
GUIs, now goes through nltk.redos on the regex engine. A pattern
longer than 100,000 characters, nested more than 100 deep, with
more than 1,000 capturing groups or an excessive repetition
product is refused with nltk.redos.error, a subclass of re.error;
a match that runs longer than nltk.redos.DEFAULT_TIMEOUT (5
seconds) raises TimeoutError instead of hanging. Patterns keep re
syntax; match objects are the regex engine's (PR #3825, #3829,
#3830, #3831, #3832, #3833, #3834, #3835, #3836, #3837, #3838,
#3839, #3840, #3841, #3842, #3843, #3844, #3845, #3846, #3796,
#3801, #3906, #3907, #3908, #3910)
- Regular expressions made linear: the leading-run and re-anchoring
patterns that a timeout cannot catch were rewritten or bounded in
the word, Treebank, blankline, TextTiling, toktok and syllable
tokenizers, the readline of corpus streams, the CoNLL SRL, TIMIT,
senseval, pl196x, NKJP, Alpino, YCOE, IEER, ACE, VerbNet, reviews
and Lin readers, relextract, the valuation parser, the CCG
lexicon parser, dependency grammars and chomsky_normal_form, each
verified on the real corpus (PR #3947, #3934, #3932, #3938,
#3935, #3933, #3869, #3944, #3911, #3909, #3953, #3895)
- Size, depth and time bounds: trees deeper than
nltk.tree.MAX_TREE_DEPTH (500), WordNet and other relation walks
deeper than nltk.util.MAX_RECURSION_DEPTH (500), tgrep deeper
than 50, Toolbox deeper than 500, dependency graphs deeper than
200, are refused with ValueError naming the limit rather than
exhausting the stack, and the stemmers, distance metrics,
skipgrams, parsers and grammar transforms run in linear or
bounded time with identical output for legitimate input.
chart_parse stops after nltk.parse.chart.DEFAULT_MAX_TIME (5
seconds of wall-clock, needed only for feature grammars;
max_time=None disables it). Tree.__repr__ no longer recurses, so
a deep accepted tree prints (PR #3827, #3885, #3895, #3767,
#3784, #3917, #3958, #3960, #3977)
- External tools and Java: a tool binary is resolved from an
absolute path or a trusted search path only, never from the
current directory; the binary, its directory and the model and
data files handed to it must not be writable by another account
(symlinked and multi-call executables such as a packaged Graphviz
dot are supported). java() accepts a main class as the first
argument only (no -jar, @argfile or .java source file), validates
its options against an allowlist, and loads jars only from a data
root unless NLTK_ALLOW_UNSAFE_JARS=1 opts out with a warning.
CoreNLP server options are allowlisted. Stanford, Malt, Senna,
hunpos, Prover9, megam, TADM, Weka and crf wrappers validate
their paths and the text they pipe to the tool against line and
argument injection (PR #3772, #3771, #3975, #3743, #3782, #3788,
#3768, #3798, #3887, #3858, #3950, #3951, #3812, #3816, #3868,
#3894, #3941, #3850, #3925)
- Network: the downloader, nltk.data.load of a URL and the CHILDES
reader fetch through pathsec.urlopen, which refuses addresses
that are not globally routable (loopback, private, link-local and
multicast, including tunnelled and obfuscated forms), pins the
validated address for the connection, allows file:// with a local
authority only, and refuses a proxied fetch unless
NLTK_ALLOW_PROXIED_URLOPEN=1 (or nltk.pathsec.ALLOW_PROXIED_FETCH
= True) asserts the proxy is trusted. The CoreNLP client talks to
its local server with requests and is not affected (PR #3847,
#3810, #3750, #3850)
- Terminal, CSV, DOT and HTML output: every print in the library
goes through nltk.termsec.safe_print, which renders control
characters, escape sequences and unbalanced bidi overrides from
corpus text as visible \xNN escapes instead of letting them reach
the terminal. The CSV files the twitter and sentiment helpers
write neutralise formula-leading and control-sequence cells,
AlignedSent DOT labels are escaped, the WordNet browser escapes
corpus text, and nltk.data.load's verbose output no longer prints
the full resource URL (PR #3914, #3915, #3937, #3946, #3931,
#3930, #3856, #3902)
- Corpus stores and encodings: the PanLex Lite and chat80 sqlite
stores open read-only on a pinned descriptor with their schema
screened and a statement step budget, chat80 shelves are read by
a key manifest, Toolbox SFM markers are sanitised, and a resource
encoding that is not a text codec (bz2_codec and friends) is
refused (PR #3961, #3964, #3959, #3888, #3800, #3962)
- Package contents: the wheel no longer ships nltk.test.unit (the
attack harnesses, advisory probes and their fixtures), so pip
install nltk is about half the size and import nltk.test.unit
works from a source checkout or the sdist only. The doctests
under nltk.test still ship (PR #3954)
Security test and CI infrastructure
- Attack and regression harnesses, advisory probes, the sandbox
conftest fixtures, the pre-commit guards (no unsandboxed open,
every regex through redos, every pickle through picklesec, every
print through safe_print), load-invariant CPU-time timing rules,
CI matrix and runner fixes, pinned third-party downloads, and the
AUTHORS update (PR #3744, #3749, #3762, #3764, #3770, #3776,
#3791, #3795, #3803, #3818, #3820, #3821, #3822, #3849, #3855,
#3860, #3861, #3862, #3864, #3865, #3866, #3876, #3884, #3899,
#3904, #3905, #3912, #3917, #3924, #3926, #3927, #3939, #3940,
#3943, #3948, #3949, #3963, #3968, #3971, #3976, #3982, #3983,
#3984, #3985)
Other changes
- Added RTL language syntactic tree visualization support (PR
#3472)
- Fix#3394: Ensure punkt_tab is downloaded when punkt is requested
(PR #3505)
- Show README.md as the PyPI long description (PR #3746)
- Add stopwords.words("english-extended") (PR #3747)
- Krippendorff's alpha: treat placeholder labels as missing data
(#2865, #2732) (PR #3752)
- Fix shared defaults in chat80 Concept (PR #3773)
- Fix Java wrappers decoding str output from java()
(universal_newlines) (PR #3777)
- Prevent IndexError in Hungarian stemmer on empty string (PR
#3785)
- Isolate MTE tag filters per view (PR #3792)
- Fix newline in ellipsis tokens in TweetTokenizer (PR #3819)
- Isolate default WordNet reference relations (PR #3872)
- Fix resolution paths that cancel distinct literals (PR #3878)
- Fix UTF-8 text preprocessing in NKJP corpus reader (PR #3879)
- Remove shadowed FeatStruct._walk stub and preserve its docstring
(PR #3893)
- Count all increasing pairs in RIBES kendall_tau (PR #3900)
- Fix the TweetTokenizer ZWJ emoji pattern capturing adjacent
whitespace (PR #3913)
- Surface the CoreNLP server's reason for a non-2xx answer (PR
#3942)
- Omit unaligned (NULL target) pairs from Alignment strings, so IBM
model output prints (PR #3967)
- Fix DependencyGraph cycle detection, which returned False on
every parsed graph (PR #3978)
- Show, range and draw an alignment with an unaligned pair on every
surface (PR #3987)
What's Changed:
- Show README.md as the PyPI long description
- Add stopwords.words("english-extended")
- Fix pathsec.urlopen treating NO_PROXY as a configured proxy
(#3748)
- Krippendorff's alpha: treat placeholder labels as missing data
(#2865, #2732)
- Harden core security modules: pathsec URL-scheme bypass,
picklesec denylist, xmlsec parse()
- Add a living per-advisory security-probe harness
- Harden tokenize file I/O against path traversal (punkt,
stanford_segmenter)
- Add a shared pathsec sandbox conftest fixture
- Contain CorpusReader / agreement / chat80 file I/O in the pathsec
sandbox
- Write parse demo/profile output to a private staging dir, not
/tmp or the CWD
- Add pathsec regression tests for sem.util / xmlsec / dataset
loaders
- Harden maxent / named-entity model-artifact saves against
shared-tmp planting
- Route weka ARFF_Formatter.write through the pathsec sandbox
- Harden untrusted-model pickle allowlist (exact globals +
numpy.lib denylist)
- Split nltk_data pickle-asset regression into always-on and
skippable tests
- Reject Java launcher switches and @argfiles in the java() cmd
argument (CWE-88)
- Prevent infinite loop from cyclic collection indices
- Fix ARFF label injection in Weka classifier
- Stabilize multiprocessing test timeouts and context handling
- Fix Java wrappers decoding str output from java()
(universal_newlines)
- Fix shared defaults in chat80 Concept
- Harden internals.java() (CWE-88): options, classpath, env,
discovery; route MaltParser through it
- Bound decompression entry points in nltk (zip/gzip layers +
pathsec.ZipFile + weka), CWE-409
- Restore pre-commit formatting and register the new probe id on
develop
- Harden pickle deserialization (picklesec) against
arbitrary-code-execution gadgets
- Fix internals.doctest to match the hardened
_validate_java_options allowlist
- Bound re_show's caller-supplied regex against ReDoS (CWE-1333)
- Refuse CWD-relative matches for bare names in find_file/find_dir
(CWE-427)
- Bound RegexpStemmer's caller-supplied pattern against ReDoS
- Deserialize chat80 shelve values through the restricted unpickler
(RCE fix)
- Harden path/resource/tool validation in nltk.pathsec
- Harden downloader Package attributes and extraction against
traversal/Zip-Slip
- Harden nltk.xmlsec.parse: read filename sources through
pathsec.open
- Validate the NE chunker save_params destination through the
tool-directory guard
- Pin maxent scratch files to a data root and validate scratch IO
- Add staging_tempdir and tighten data.py zip/staging guards
- Stage boxer scratch input inside an nltk.data root
- Canonicalize obfuscated numeric IPv4 in the SSRF guard (CWE-918)
- Harden the VERSION read in nltk package init with
pathsec.open_package_resource
- Harden stanford tokenizer/segmenter wrappers: validate jar/model
paths and argv
- Harden tagger model loads and tool paths
(perceptron/crf/hunpos/stanford)
- Stage weka scratch files in a data root and validate the WEKAHOME
jar path
- Route corpus-reader scratch tempfiles through a data root and
label framenet path refusals
- Harden malt and stanford parser wrappers: validate
binary/model/IO paths and argv
- Harden nltk.tbl.demo model load/save through picklesec and
pathsec
- Add tests covering already-merged pathsec/security guards
- Fix develop CI: register data roots for benign save/IO tests
broken by pathsec hardening
- Prevent combinatorial DoS in skipgrams
- Prevent IndexError in Hungarian stemmer on empty string
- Sanitize SFM markers in ToolboxData
- Isolate MTE tag filters per view
- Prevent classpath shadowing in GenericStanfordParser
- Prevent batch-boundary injection in StanfordParser
- Add remaining GHSA-8mgp security regression tests
(pathsec/java/redos/archive/corpus)
- Add redos.source_of/reharden and picklesec.harden_object_graph
- Load nltk.tbl.demo models through an allowlisting unpickler
- Bound algorithmic-complexity (time) DoS in stemmers, metrics,
parsers (CWE-407 / CWE-770)
- Bound regex compilation against a compile-time DoS
- Route top-level nltk regexes through nltk.redos
(CWE-1333/CWE-400)
- Route nltk.sentiment regexes through nltk.redos
(CWE-1333/CWE-400)
- Route nltk.chat regexes through nltk.redos (CWE-1333/CWE-400)
- Route nltk.corpus regexes through nltk.redos (CWE-1333/CWE-400)
- Route nltk.parse regexes through nltk.redos (CWE-1333/CWE-400)
- Route nltk.draw regexes through nltk.redos (CWE-1333/CWE-400)
- Route nltk.stem regexes through nltk.redos (CWE-1333/CWE-400)
- Route nltk.translate regexes through nltk.redos
(CWE-1333/CWE-400)
- Route nltk.sem regexes through nltk.redos (CWE-1333/CWE-400)
- Route nltk.tree regexes through nltk.redos (CWE-1333/CWE-400)
- Route nltk.tokenize regexes through nltk.redos (CWE-1333/CWE-400)
- Route nltk.chunk regexes through nltk.redos (CWE-1333/CWE-400)
- Route nltk.classify regexes through nltk.redos (CWE-1333/CWE-400)
- Route nltk.tag regexes through nltk.redos (CWE-1333/CWE-400)
- Route nltk.app regexes through nltk.redos (CWE-1333/CWE-400)
- Route nltk.ccg regexes through nltk.redos (CWE-1333/CWE-400)
- Bound compile-time DoS (ValueError) in the concordance and nemo
GUI regex handlers (CWE-1333/CWE-400)
- Harden RegexpStemmer against pre-compiled ReDoS patterns
(CWE-1333)
- Add security regression tests + harden SSRF filter (transition
tunnels, IP literals)
- Strengthen the advisory security-probe harness (guards already on
develop)
- Guard model-artifact loaders against un-sandboxed open()
- Route security-test fixtures through the pathsec I/O helpers
- Stage REPP scratch input in an nltk.data root instead of the
shared system temp
- Bound JSON deserialization against deep-nesting crash and
resource DoS
- Fix newline in ellipsis tokens in TweetTokenizer
- Refuse NUL bytes in pathsec.validate_path; add a pathsec
attack-candidate harness
- Re-check the normalized form of a no-protocol resource path
against traversal (CWE-22)
- Add and expand security / ReDoS attack test coverage
- Escape corpus-derived text in the WordNet browser app (CWE-79
XSS)
- Add test-only regression coverage for already-fixed CWEs
- Extend the no-unsandboxed-open guard to app/draw/twitter and
route every open through pathsec.open
- Add the all-regex-through-redos pre-commit guard and two
regression tests
- Exclude pytest's ephemeral basetemp from the pickle-asset scanner
- Open operator-chosen app/twitter paths with an annotated builtin
open, not pathsec.open
- Bound the WekaClassifier model path to the pathsec data roots
- Tidy transitionparser security comments to one point per <=3
lines
- Deflake anti-DoS timing tests and harden senseval _fixXML
leading-run ReDoS
- Harden and standardize NLTK's external-tool wrappers on a pathsec
trusted-spawn API (CWE-426/427/732/93/88)
- Don't miscount a python-crfsuite Windows failure as a pathsec
over-block (tag sweep)
- Give every pathsec refusal the standard Security Violation marker
(spawn_trusted + ZIP audit)
- Bound recursion depth in parsers and traversal helpers
- Isolate default WordNet reference relations
- Harden and complete the CoreNLP server-option allowlist
(CWE-88/22/502)
- Optimize CI test matrix
- Harden CoreNLPServer start attempts
- Make DoS/ReDoS timing regression tests load-invariant (fix CI
flakiness)
- Extend no-unsandboxed-open guard to
gzip/bz2/lzma/zipfile/tempfile (+ coverage test)
- Count all increasing pairs in RIBES kendall_tau
- Potential fix for code scanning alert no. 28: Clear-text logging
of sensitive information
- Remove shadowed FeatStruct._walk stub and preserve its docstring
- Fix resolution paths that cancel distinct literals
- Add GHSA-j456 live probe; expand the GHSA-8mgp probe attack set
- Handle the redos MAX_PATTERN_LENGTH refusal in
help._format_tagset
- Handle the redos MAX_PATTERN_LENGTH refusal in
TokenSearcher.findall
- Handle the redos MAX_PATTERN_LENGTH refusal for a tgrep /regex/
node literal
- Make help._format_tagset fail closed on an oversized tag pattern
(fix develop CI)
- Add nltk.termsec terminal-output sanitizer with attack-matrix
tests
- [WIP] Fix quadratic DoS in Tree.chomsky_normal_form right-factor
binarization (GHSA-r53h-rw34-8h97)
- Bound SyllableTokenizer fallback-vowel accumulation (CWE-400 /
CWE-407)
- Fix develop CI on Python 3.10/3.11: let deepcopy reach
MAX_TREE_DEPTH
- Fix the XMLCorpusView ReDoS-test Windows CI flake (in-process op
timing + Windows autoload)
- Route every library print through the nltk.termsec.safe_print
chokepoint
- Log the perceptron training progress on the module logger (fixes
the develop 3.10/3.11 cells)
- Resolve every executed tool binary absolute-only: prover9/mace4,
megam, tadm, java, hunpos (untrusted search path, CWE-426/427)
- Pin the GHSA-cc5r-64rf-75hg proof of concept end to end and run
its sink as probe phase 6 (follow-up to #3887)
- Validate chat80.sql_query dbname for path containment (CWE-59/73)
- Extend the tool model-file guards to crf, stanford tagger and
segmenter
- Consolidate the internals, downloader, pathsec and decorators
guards from #3753 and #3850
- Consolidate the nltk/parse guards from #3753 and #3850 and judge
their inputs on real characters
- Sanitise the CSV cells and report lines the sentiment helpers
write
- Read a long line in linear time in
SeekableUnicodeStreamReader.readline (GHSA-j8g8)
- Escape AlignedSent DOT labels, refuse NUL, and keep the alignment
checks under -O
- Align quote tokens in linear time in the Treebank and NLTK word
tokenizers
- Never leave the pop of an escaped bidi override live in
sanitize_terminal
- Find readline's line boundary with str.splitlines, not a timed
regex
- Walk semi-relations once in relextract, and keep a pattern's
flags in redos.reharden
- Fix UTF-8 text preprocessing in NKJP corpus reader
- Make the CoNLL SRL, TIMIT and XML view readers linear, and parse
corpus XML through xmlsec (GHSA-v8f3)
- Measure probe scaling in process CPU time so a loaded runner
cannot flip a verdict
- ci(deps): bump dorny/paths-filter from 3 to 4 in the
github-actions group[bot] in
https://github.com/nltk/nltk/pull/3943
- Surface the CoreNLP server's reason for a non-2xx answer and warm
the live test server
- Guard the Stanford tagger and segmenter input against line
injection, verified on the real tools
- Bring the self-contained tests of #3753 to develop
- Guard the twitter CSV exports against formula and
control-sequence injection, verified on the real corpus
- Guard the maxent parameter files against outside-root paths and
unrepresentable names, verified on a real trained model
- Parse CCG lexicon categories in linear time and refuse oversized
ones, verified under attack and on the documented grammars
(GHSA-89p3-fcch-88ph, split from #3753)
- [Do not merge yet] Guard tool wrappers, Prover9 input and the
downloader against injection and unbounded-response DoS
- Added RTL language syntactic tree visualization support
- Make the word, blankline and TextTiling tokenizers, the valuation
parser, relextract and toktok linear on retried leading regex
runs
- Add the 3.10.4 commit authors and security reporters to
AUTHORS.md
- Pair the scaling samples by rep and size the under-margined
probes over the floor
- Trusted spawn: O_NOFOLLOW and inode check on the final file, link
owner and control-character checks, refusal reasons, argv[0] as
invoked; real Graphviz in CI (#3887 follow-up)
- Support symlinked executables in pathsec
- [Do not merge yet] Keep downloaded archives private and serve
other accounts through extraction (fixes#3928)
- Bound algorithmic-DoS and uncontrolled-recursion vectors
(CWE-400/407/674/1333)
- Guard the two-class any-character runs and count apostrophe-named
groups (follow-up to #3895)
- Fix#3394: Ensure punkt_tab is downloaded when punkt is requested
- Restore add_blank_lines and to_settings_string after the
GHSA-f2h2 depth bound
- Let chat80.val_load read back what val_dump wrote
- Keep full collections out of timed scaling samples (89p3 teeth
and relextract reds)
- CI: take the Windows Graphviz from the cached third-party tools,
not Chocolatey
- Close the chat80 store swap window and read stores by a key
manifest
- Route every module's file removal, rename, privacy test and
sqlite open through pathsec
- Fix the develop CI breakage from the fg9w advisory merge (Senna
-path separator, test fixtures under /tmp)
- Refuse relation walks deeper than MAX_RECURSION_DEPTH instead of
truncating them
- Harden the Toolbox reader and writers after the #3958 re-audit
- Route the downloader, XML corpus view, internals and Toolbox
through nltk.xmlsec and nltk.data
- Open gzip file names through pathsec, and refuse a linked PanLex
Lite database
- Fix DependencyGraph cycle detection, dead on every parsed graph
- Bound YAML deserialization against depth and resource DoS,
verified on the real loaders
- Guard java() cmd[0] against JEP 330 source-file execution,
verified on a real JDK
- Make Tree.__repr__ iterative so a deep accepted tree does not
overflow the stack
- Create perceptron model files exclusively under the pinned
directory (issue 3980)
- Drop the dash-banner section dividers from the model-artifact
tests
- Skip the real-launcher test when java resolves to a runtime-less
stub
- Pin the CI megam download to https and its sha256
- Run the advisory probes only on a full-matrix run; keep the unit
tests out of the wheel
- Charge the shipped-form teeth leg on CPU time (Windows regex
timeout is wall-clock)
- Prepare NLTK release 3.10.4
- Rewrite the 3.10.4 release notes around the user-facing security
changes
- Omit NULL targets from alignment strings
- Fix TweetTokenizer ZWJ pattern capturing whitespace
- Show, range and draw an alignment with an unaligned pair on every
surface (follow-up to #3967, #1885)
- Pin the TweetTokenizer ZWJ alternative: linear on repeated
shapes, no whitespace absorbed (follow-up to #3913)
- Credit every contributor since 3.10.3 in the 3.10.4 notes and
AUTHORS.md; name the PRs merged after #3986
- Make the span_tokenize teeth control compute-bound so Windows
reads its quadratic (deflake, test-only)
Signed-off-by: Leon Anavi <leon.anavi@konsulko.com>
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
Upgrade to release 9.1.0:
- Removed support for Python 3.10, now end-of-life.
- Where a path supplies more than one metadata directory for a
distribution, as an interrupted or unclean upgrade may leave
behind, discovery now prefers the greater one by reverse
lexicographic ordering of the directory names, so the stale
metadata no longer shadows the version in use.
- A missing required metadata field (Name or Version) now raises a
KeyError that names the field and identifies the distribution,
which PathDistribution renders as its metadata path.
- Restored lazy validation of EntryPoint.value, deferring the
ValueError for an invalid value until a component is accessed or
the entry point is loaded, so one malformed entry point no longer
prevents entry points in other groups from being selected.
- importlib_metadata.diagnose no longer crashes on a metadata
directory that has no metadata file or whose metadata declares no
name. Such distributions are now reported by their location.
Signed-off-by: Leon Anavi <leon.anavi@konsulko.com>
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
Upgrade to release 2.5.0:
- imgtool now requires Python 3.8 or later. This also fixes key
export commands, which failed on Python 3.8 and 3.9.
- Added imgtool keyinfo, which reports whether a PEM file holds a
key pair or only a public key. With --require, it fails if the
kind does not match, which build systems can use as a check.
- Added --name-suffix to imgtool getpub and imgtool getpubhash,
which appends a suffix to the generated C or Rust symbol names so
that several keys can be embedded in one image.
- Public-key-only PEM files are now supported by imgtool getpub,
imgtool getpubhash and imgtool verify. imgtool sign given a
public-only PEM now fails with a clear error.
Signed-off-by: Leon Anavi <leon.anavi@konsulko.com>
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
Leon Anavi
Khem Raj
Ankur Tyagi
Patrick Wicki
Ross Burton
Jason Schonberg
Ricardo Salveti
Khem Raj