Upgrade to release 1.2.3:
- Restored the ability to import
pymysql.converters.escape_bytes_prefixed for compatibility
with aiomysql.
- Use pymysql.converters at your own risk. It's internal functions.
No backward compatibility are guaranteed.
Signed-off-by: Leon Anavi <leon.anavi@konsulko.com>
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
Upgrade to release 4.19.0:
- Added support for Python 3.15.
- Dropped support for Python 3.9 and 3.10.
- Removed support for PyPy.
- Removed the deprecated Algorithm.RANGEPREVIEW and
QueryType.RANGEPREVIEW members (PYTHON-6009).
- Added support for running the synchronous and asynchronous
clients in subinterpreters, including inside
concurrent.futures.InterpreterPoolExecutor (Python 3.14+). Only
in subinterpreters, which do not allow daemon threads, the
synchronous client's monitor threads now start as non-daemon
threads and are stopped and joined when the interpreter is torn
down. Monitor threads in the main interpreter remain daemon
threads and shutdown behavior is unchanged. Note that because
these threads are non-daemon, a subinterpreter may block on
teardown until any in-flight monitor work completes.
- Added support for routing Key Management Service (KMS) requests
for Client-Side Field Level Encryption and Queryable Encryption
through an HTTP proxy, using the new kms_connect_callback option
on AutoEncryptionOpts, ClientEncryption, and AsyncClientEncryption.
The callback opens the connection and the driver performs the KMS
TLS handshake over it, so verification still targets the KMS host
rather than the proxy. For an ordinary HTTP proxy, pass
HTTPProxyKMSConnect or AsyncHTTPProxyKMSConnect instead of writing
a callback. Both helpers accept custom CONNECT headers, e.g.
Proxy-Authorization for proxies that require authentication.
- Added the srv_host_validator keyword argument to MongoClient and
AsyncMongoClient, an alternative to srvAllowedHostsSuffix for
deployments whose acceptable SRV hosts cannot be expressed as a
single suffix. The callback is invoked once per SRV-returned host
and must return a boolean: True to accept the host and False to
reject it. During initial DNS discovery a non-boolean return
raises an error; during SRV polling it is treated as a rejection.
It is mutually exclusive with srvAllowedHostsSuffix and, because
it takes a callable, cannot be set in a connection string. See
the MongoClient and AsyncMongoClient documentation for security
considerations.
- srvAllowedHostsSuffix may now be set to a single label reserved
for private or special use, such as localhost, test, or internal.
Signed-off-by: Leon Anavi <leon.anavi@konsulko.com>
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
Upgrade to release 2.15.1:
- Accept trailing Base64URL = padding when decoding JWS segments,
so tokens issued by AWS ALB and similar systems verify instead of
raising DecodeError: Invalid crypto padding. Non-alphabet junk
such as !!!! remains rejected
Signed-off-by: Leon Anavi <leon.anavi@konsulko.com>
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
Upgrade to release 16.0.0:
- BREAKING: Use the Self type introduced in Python 3.11
Error handling improvements:
- Avoid logging callback failures when queued executor handlers
are canceled
Documentation improvements:
- Add Args, Returns and Raises sections to public docstrings
- AGENTS.md file for AI coding assistant guidance
Local development changes:
- Type check with both pyright and mypy by default
- Remove tox test matrix, leaning on Github Actions
Signed-off-by: Leon Anavi <leon.anavi@konsulko.com>
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
Upgrade to release 7.48.0:
- Expose more CURLOPT_SSL_OPTIONS (patch by Scott Talbert)
- Read long description from README.rst instead of duplicating it
in setup.py (patch by Scott Talbert)
- Fix test_redir_protocols_setopt (patch by Scott Talbert)
- Add CURL_LOCK_DATA_HSTS as a CurlShare option (patch by Scott
Talbert)
- Add libpsl support to vcpkg installs for CI and wheel builds
(patch by Scott Talbert)
- Configure nuget repo access as read-only from PRs (patch by Scott
Talbert)
- Officially change project URL to https://pycurl.github.io/ (patch
by Scott Talbert)
- Replace bare except clauses in examples with specific exception
types (patch by Kotesh Kumar Yelamati)
- Add HTTP/3 support in libcurl wheel (patch by Will Noble)
- Fix segfault when calling duphandle() on a closed Curl handle
(patch by Jorge Rocamora)
- Fix abort on non-text string arguments (patch by Jorge Rocamora)
- Fix CurlShare leak when close() detaches attached handles (patch
by Jorge Rocamora)
- Document that views derived from the callback memoryview are not
released (patch by Jorge Rocamora)
- Fix crash when duplicating a handle that uses set_ca_certs()
(patch by Jorge Rocamora)
- Add CurlUrl object wrapping libcurl's URL API, with CURLOPT_CURLU
support (patch by Jorge Rocamora)
- Fix callback exceptions being replaced by SystemError (patch by
Jorge Rocamora)
- Fix test guard silently passing on real libcurl errors (patch by
Jorge Rocamora)
- Update the docstring and manifest source lists (patch by Jorge
Rocamora)
- Refuse pause() from a thread other than the performing one (patch
by Jorge Rocamora)
- Add support for Python 3.15 (patch by Scott Talbert)
- Refuse integers for blob options instead of crashing (patch by
Jorge Rocamora)
- Accept buffer objects for blob options (patch by Jorge Rocamora)
- Release all httppost encoded values and unset the form before
MIMEPOST (patch by Jorge Rocamora)
- Reject callback return values that do not fit in an int (patch by
Jorge Rocamora)
- Clear CurlMulti callbacks before deallocating attached handles
(patch by Eric Buehl)
- Serve the test apps from a threaded WSGI server (patch by Jorge
Rocamora)
- Add support for missing CURL options (patch by Khavish Anshudass
Bhundoo)
- Add CURLOPT_SSL_EC_CURVES & CURLOPT_SSL_SIGNATURE_ALGORITHMS
options (patch by Khavish Anshudass Bhundoo)
- Remove unused variable from do_curl_setopt_httppost (patch by
Scott Talbert)
- test: silence WSGI test server access logging to stderr (patch by
Scott Talbert)
- test: pin down oversized mime data_cb error assertions (patch by
Scott Talbert)
- Give vendored OpenSSL a private OPENSSLDIR on Linux manylinux
wheels (patch by Scott Talbert)
- Enable -Werror for the pycurl build in cibuildwheel without
affecting vcpkg (patch by Scott Talbert)
- Handle libcurl options deprecated in 8.22.0 (patch by Scott
Talbert)
Signed-off-by: Leon Anavi <leon.anavi@konsulko.com>
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
Upgrade to release 2.15.0:
- Improve performance of module_imports_on_top_of_file.
- Improve performance of python_3000_invalid_escape_sequence.
- Improve performacne of missing_whitespace_after_keyword.
- Drop EOL python 3.9.
- Add support for python 3.15.
- Fix loading unrelated configuration due to commonprefix.
Signed-off-by: Leon Anavi <leon.anavi@konsulko.com>
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
Upgrade to release 7.36.2:
- Detects single-crate cross-file symbol collisions and
conditionally emits crate-root re-exports.
- Prepend the file module name to Rust type paths to use canonical
paths.
- Expose generated per-file Rust proto modules as public.
- Include .c files in the source_files glob in upb/port/BUILD
Signed-off-by: Leon Anavi <leon.anavi@konsulko.com>
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
Upgrade to release 2.6.3:
- IntervalDict.update and __init__ behave like dict.update when the
provided intervals overlap (the last provided value wins).
- IntervalDict.pop accepts None as default value.
- Fix from_string accepting prefix (re.match) instead of full value
(re.fullmatch) for infinities.
- Documentation of iterate explicitly states that base should be
side-effect-free.
Signed-off-by: Leon Anavi <leon.anavi@konsulko.com>
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
Upgrade to release 1.22.4:
- Account for the capture groups %z adds to the generated pattern
- Reject out-of-range timezone minutes
- Document the extra_types conversion mapping
- Fix case-sensitive am/pm and Zulu conversion under
case-insensitive matching
- Resolve year-less strftime formats against the current year
Signed-off-by: Leon Anavi <leon.anavi@konsulko.com>
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
Upgrade to release 2.17:
- Added support for DateTimePatternGenerator.getDefaultHourCycle()
and enums
- Added missing wrappers for UDateFormatField
- Added support for ICU 79.1
Signed-off-by: Leon Anavi <leon.anavi@konsulko.com>
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
Upgrade to release 3.5.3:
- Close the storage connection before forking process workers, this
just simplifies things.
- Run the callback of an empty chord with an empty results list.
Previously it was never enqueued.
Signed-off-by: Leon Anavi <leon.anavi@konsulko.com>
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
Upgrade to release 4.3.1:
- Fix type-check failures with ty 0.0.87
- Exclude .DS_Store from release
- Route annotated command help through the application output
stream
- Add example embedding a cmd2 app in a Textual app with
textual-tty
- Add example embedding a cmd2 app in a Textual app with
textual-tty
Signed-off-by: Leon Anavi <leon.anavi@konsulko.com>
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
Upgrade to release 7.2.5:
- Fix @cachedmethod with condition for decorated methods taking a
pending keyword argument (broken since v7.0.0).
- Fix pickling and deep-copying of instances using @cachedmethod
(broken since v7.0.0).
- Fix calling @cachedmethod decorated methods through the class
with the receiver (self) passed as a keyword argument (broken
since v7.2.4).
- Various documentation improvements.
Signed-off-by: Leon Anavi <leon.anavi@konsulko.com>
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
Upgrade to release 3.13.0:
- It is now possible to nest ThreadSensitiveContext by providing an
explicit executor for thread-sensitive tasks. This enables moving
a block of such work away from the parent context's shared worker.
The provided executor must run work sequentially and on a single
thread, as per ThreadPoolExecutor(max_workers=1), and the calling
code is responsible for managing the executor's lifecycle.
- Local attribute access is now 2-3x faster. Local(...) now uses one
of two internal specialisations, selected by thread_critical,
instead of checking the mode on every access.
As a result Local is now marked @final and raises TypeError if
subclassed. Subclassing was never supported, and could not have
worked correctly with the specialised implementations.
- asgiref.timeout.timeout is deprecated, and will be removed in a
future release. Move to use asyncio.timeout() instead.
- Added support for Python 3.15.
- Dropped support for EOL Python 3.10.
Signed-off-by: Leon Anavi <leon.anavi@konsulko.com>
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
pyproject.toml specifies setuptools.build_meta as the build backend, so
inherit python_setuptools_build_meta instead of setuptools3.
Fixes:
QA Issue: inherits setuptools3 but has pyproject.toml specifying backend
setuptools.build_meta, use the correct class [pep517-backend]
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
- Recognize timezone file names with multiple slashes in the TZ environment
variable.
- Use HTTPS in place of FTP to retrieve tzdata archive.
Signed-off-by: Ankur Tyagi <ankur.tyagi85@gmail.com>
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
simple-websocket is the only hard dependency in pyproject.toml, and the
default threading async mode uses it for WebSocket connections. The
threading driver and the client also import queue (python3-threading),
and the client imports ssl (python3-io).
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
Simple WebSocket server and client for Python, built on wsproto.
Backport the upstream conversion of the async tests to pytest-asyncio;
the 1.1.0 tests call asyncio.get_event_loop(), which fails on Python
3.14 when there is no current event loop. Add it to the fast ptest list;
all 47 tests pass on qemuriscv64.
AI-Generated: Uses Claude Code
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
Changelog:
* Drop Python 2 support; require Python 3.5+
* Remove the module-level names INT_TYPES, STR_TYPES and xrange
* Add inline type annotations; IPy is now a package that ships py.typed
* Fix IP.broadcast() returning an IPv4 address for IPv6 networks
* Fix IP.reverseName() for 0.0.0.0/0
* Fix the flaky IPSet fuzz test
The fuzz test fix stops the intermittent ptest failure in
TestIPSet.testRandomContains, whose oracle ignored that IPSet merges
adjacent prefixes into a larger one.
The sdist is now named ipy-1.2.tar.gz, so drop PYPI_PACKAGE_SDIST.
AI-Generated: Uses Claude Code
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
The test expects a message sent from another thread 0.2s later to arrive
within a 0.5s poll window, which a loaded test host misses. Upstream
marks it flaky(reruns=3), but pytest-rerunfailures is not available.
Adjust the musl sed to the new end of the -k expression.
AI-Generated: Uses Claude Code
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
test_async_cat_file_ranges fetches a file from a public S3 bucket, which
fails without DNS and internet access on the target, as in sysvinit
qemu images where resolv.conf is empty.
AI-Generated: Uses Claude Code
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
Upgrade to release 10.17.0:
- Added {class}pikepdf.ObjectSet, which records the objects that a
walk of a PDF's object graph has already visited. Objects in a
PDF are shared and may refer back to their ancestors, so a search
that only limits its depth can take exponential time on quite
ordinary files. ObjectSet.add() reports whether an object is new,
and tells apart direct objects and objects of different Pdf,
which a set of objgen cannot. See {ref}walking-object-graphs.
- Added {meth}pikepdf.Object.is_same_object_as, which tests whether
two references lead to the same object. Neither is, == nor
comparing objgen answers this reliably.
- Added {meth}pikepdf.Pdf.content_scopes and
{meth}pikepdf.Page.content_scopes, which iterate over every
content stream that pages can draw, each with the resource
dictionary it uses: the page itself, form XObjects, annotation
appearance streams, tiling patterns, Type 3 glyph procedures and
soft masks, however they are nested or shared. This is the way to
find every font, image or colour space a document uses. Each
{class}pikepdf.ContentScope is reported once for the document;
pages= selects pages and kinds= selects what is reported.
- Added {meth}pikepdf.Page.get_resources, which returns the
resources of a page or form XObject for reading. Unlike
{attr}pikepdf.Page.resources, it finds resources inherited from
the page tree, does not create an empty dictionary when there are
none, and returns None instead of raising when /Resources is
malformed.
- Fixed comparing two objects with == taking exponential time when
the objects being compared share children, such as the same file
opened twice. Each pair of objects is now compared once.
- Fixed {meth}pikepdf.Pdf.add_pages_from hanging on a page with an
action whose /Next actions are shared or form a loop.
- Fixed {meth}pikepdf.Pdf.open_outline reading only the first item
at each level of an outline whose items are direct objects. Such
items are now made indirect when the outline is saved, as the PDF
specification requires.
- Reading a stream that will not decode now raises
{exc}~pikepdf.exceptions.DataDecodingError with the reason,
whether the stream's data was read from a file or set from
memory. Previously a stream read from a file raised a
{exc}~pikepdf.exceptions.PdfError that said only "read_bytes
called on unfilterable stream", and the reason was available only
from {meth}pikepdf.Pdf.get_warnings. DataDecodingError is a
PdfError, so existing handlers are unaffected.
- An exception raised by a JBIG2 decoder now reaches the caller of
{meth}pikepdf.PdfImage.as_pil_image,
{meth}pikepdf.PdfImage.extract_to and
{meth}pikepdf.PdfImage.read_bytes unchanged, with its own type,
message and traceback. Previously, for an image read from a file,
it was replaced by a generic DataDecodingError or PdfError.
- Added {exc}~pikepdf.exceptions.QpdfRuntimeError, raised for
errors from qpdf that pikepdf has no more specific exception for.
These were previously raised as a bare RuntimeError, which except
pikepdf.PikepdfError did not catch. The new exception derives
from both {exc}~pikepdf.exceptions.PikepdfError and RuntimeError,
so handlers written for RuntimeError still work.
- The empty name / is now supported. The PDF 1.7 and 2.0
specifications define a lone slash as a valid name made of zero
characters, but pikepdf.Name('/') raised ValueError, and
dictionaries refused / as a key. This broke
{meth}pikepdf.Pdf.add_pages_from on files with an empty-named
destination such as /Dests << / 3 0 R >>. Such names can now be
created, used as dictionary keys, and migrated during page
copying. bool(Name('/')) is False, like an empty
{class}pikepdf.String. Name(''), which lacks the leading slash,
is still rejected. PDF/A does not forbid empty names, and the
PDF/A validator accepts them.
- AcroFormField.get_inheritable_field_value_as_name() now returns a
{class}pikepdf.Name, as documented, instead of a str. When the
value is missing or not a name, it returns the empty name
Name('/'), which is falsy, instead of ''. Comparisons such as ==
'/Tx' are unaffected.
- Fixed an out-of-bounds read when opening a PDF from a Python file
object whose readinto() reports more bytes than were requested.
This now raises ValueError, as writing to such a stream already
did. Non-blocking streams, whose readinto() returns None, now
raise ValueError instead of hanging.
- Fixed building from source on PyPy, which failed to compile since
v10.0. PyPy now reads Python file objects through readinto(),
like CPython, instead of copying the result of read().
- {func}pikepdf.open reads directly from the file descriptor when
given a filename, instead of calling the seek() and readinto()
methods of a Python file object for every read. Opening and
reading a large file with many objects is about four to five
times faster, matching the speed of memory mapping
(access_mode=AccessMode.mmap). Streams passed to
{func}pikepdf.open, including plain file objects, are read
through their methods as before.
- Documented {class}pikepdf.TokenFilter with worked examples that
shorten numbers and convert RGB colors to grayscale, and
explained when to use a token filter instead of
{func}pikepdf.parse_content_stream. The API reference no longer
says that the parser should not be used to edit content streams.
The TokenFilter.handle_token documentation now says correctly
that exceptions it raises propagate to the caller.
Extend RDEPENDS with the missing runtime dependencies
python3-fonttools and python3-jsonschema which are required
to import pikepdf.pdfa.
Install pdfa_differential.py to pass the tests from
test_pdfa_differential.py. Fixes:
E ModuleNotFoundError: No module named 'pdfa_differential'
This work was sponsored by GOVCERT.LU.
Signed-off-by: Leon Anavi <leon.anavi@konsulko.com>
Signed-off-by: Khem Raj <raj.khem@gmail.com>
Upgrade to release 3.10.4:
This release routes every file path, archive, download, URL fetch and
external program through one sandbox that keeps reads and writes inside
the trusted data directories, refuses path traversal, symlink escapes
and decompression bombs, blocks requests to internal network addresses,
and runs Java and other third-party tools only from trusted absolute
locations with validated options and input. Pickled models now load
through a restricted unpickler, XML and JSON parsing are hardened, and
regular expressions, recursion and slow algorithms in the tokenizers,
parsers, stemmers and corpus readers are bounded against
denial-of-service input. Text that NLTK writes to the terminal, to CSV
files, to the WordNet browser and to third-party tool input is sanitized
against injection.
Security changes you may notice
Every refusal below raises with a message that starts with "Security
Violation" and names the check, so a failure is easy to search for (PR
- Data directories (nltk.pathsec, nltk.data): corpora, models and
every other resource are read and written only inside the
directories listed in nltk.data.path. A path that leaves a data
root, a symlink that points out of one, and a NUL byte or encoded
traversal in a resource name are refused. A data root, or a
directory on the way to a resource, that another account can
write to is skipped with a warning naming the directory and the
remedy. Keep private data in a private directory and add it to
nltk.data.path; /tmp is not a data root. Model save and load
methods (perceptron, maxent, named-entity chunker, Weka, crf,
hunpos, Stanford) accept destinations inside a data root only (PR
#3797, #3753, #3850, #3761, #3755, #3759, #3760, #3763, #3806,
#3807, #3808, #3809, #3813, #3814, #3815, #3848, #3851, #3852,
#3853, #3863, #3871, #3888, #3945, #3961, #3969, #3981)
- Downloader: downloaded package archives stay private to the
installing account (0600). An install another account must read
is extracted: a download made as root or outside the installing
account's home extracts on its own, and nltk.download(...,
extract=True) / python -m nltk.downloader --extract (or
--no-extract) decide explicitly. This also unpacks the packages
the index leaves zipped (wordnet, omw-1.4 ...); the extracted
files are readable by every account and are found before the
archive. find() names an archive it cannot read and says how to
extract it instead of reporting it "not found". The downloader
installs only into directories no other account can write to,
treats an archive another account owns or can write to as stale
and downloads it afresh, extracts only from an archive no other
account could have changed since its checksum was verified,
bounds the size of what it fetches, and refuses index entries and
archive members that would write outside the package directory
(PR #3929, #3804, #3850, #3778)
- Pickles: nltk.data.load(..., format="pickle") reconstructs plain
data only; a pickle that names any class or function is refused.
Model loaders that need classes use an explicit allowlist of
audited globals, so a gadget such as os.system can never be
reconstructed from a model file. chat80 shelve values load the
same way (PR #3794, #3754, #3757, #3802, #3817, #3823, #3824)
- XML, JSON and YAML: nltk.xmlsec refuses any document that
declares an entity or references an external one (billion-laughs,
external-entity and DTD attacks) and bounds the tree depth, and
every corpus XML view, the downloader index, Toolbox and
internals parse through it. JSON resources load with a depth and
size bound; YAML (the markdown corpus front matter and
nltk.data.load(..., format="yaml")) loads through a bounded safe
loader. Oversized or too deeply nested input raises ValueError
naming the limit (PR #3754, #3805, #3828, #3933, #3972, #3974)
- Regular expressions (nltk.redos): every regular expression NLTK
compiles, including one you pass to RegexpTokenizer,
RegexpStemmer, RegexpParser, re_show, TokenSearcher.findall,
tgrep, nltk.help tag-set lookups and the concordance and nemo
GUIs, now goes through nltk.redos on the regex engine. A pattern
longer than 100,000 characters, nested more than 100 deep, with
more than 1,000 capturing groups or an excessive repetition
product is refused with nltk.redos.error, a subclass of re.error;
a match that runs longer than nltk.redos.DEFAULT_TIMEOUT (5
seconds) raises TimeoutError instead of hanging. Patterns keep re
syntax; match objects are the regex engine's (PR #3825, #3829,
#3830, #3831, #3832, #3833, #3834, #3835, #3836, #3837, #3838,
#3839, #3840, #3841, #3842, #3843, #3844, #3845, #3846, #3796,
#3801, #3906, #3907, #3908, #3910)
- Regular expressions made linear: the leading-run and re-anchoring
patterns that a timeout cannot catch were rewritten or bounded in
the word, Treebank, blankline, TextTiling, toktok and syllable
tokenizers, the readline of corpus streams, the CoNLL SRL, TIMIT,
senseval, pl196x, NKJP, Alpino, YCOE, IEER, ACE, VerbNet, reviews
and Lin readers, relextract, the valuation parser, the CCG
lexicon parser, dependency grammars and chomsky_normal_form, each
verified on the real corpus (PR #3947, #3934, #3932, #3938,
#3935, #3933, #3869, #3944, #3911, #3909, #3953, #3895)
- Size, depth and time bounds: trees deeper than
nltk.tree.MAX_TREE_DEPTH (500), WordNet and other relation walks
deeper than nltk.util.MAX_RECURSION_DEPTH (500), tgrep deeper
than 50, Toolbox deeper than 500, dependency graphs deeper than
200, are refused with ValueError naming the limit rather than
exhausting the stack, and the stemmers, distance metrics,
skipgrams, parsers and grammar transforms run in linear or
bounded time with identical output for legitimate input.
chart_parse stops after nltk.parse.chart.DEFAULT_MAX_TIME (5
seconds of wall-clock, needed only for feature grammars;
max_time=None disables it). Tree.__repr__ no longer recurses, so
a deep accepted tree prints (PR #3827, #3885, #3895, #3767,
#3784, #3917, #3958, #3960, #3977)
- External tools and Java: a tool binary is resolved from an
absolute path or a trusted search path only, never from the
current directory; the binary, its directory and the model and
data files handed to it must not be writable by another account
(symlinked and multi-call executables such as a packaged Graphviz
dot are supported). java() accepts a main class as the first
argument only (no -jar, @argfile or .java source file), validates
its options against an allowlist, and loads jars only from a data
root unless NLTK_ALLOW_UNSAFE_JARS=1 opts out with a warning.
CoreNLP server options are allowlisted. Stanford, Malt, Senna,
hunpos, Prover9, megam, TADM, Weka and crf wrappers validate
their paths and the text they pipe to the tool against line and
argument injection (PR #3772, #3771, #3975, #3743, #3782, #3788,
#3768, #3798, #3887, #3858, #3950, #3951, #3812, #3816, #3868,
#3894, #3941, #3850, #3925)
- Network: the downloader, nltk.data.load of a URL and the CHILDES
reader fetch through pathsec.urlopen, which refuses addresses
that are not globally routable (loopback, private, link-local and
multicast, including tunnelled and obfuscated forms), pins the
validated address for the connection, allows file:// with a local
authority only, and refuses a proxied fetch unless
NLTK_ALLOW_PROXIED_URLOPEN=1 (or nltk.pathsec.ALLOW_PROXIED_FETCH
= True) asserts the proxy is trusted. The CoreNLP client talks to
its local server with requests and is not affected (PR #3847,
#3810, #3750, #3850)
- Terminal, CSV, DOT and HTML output: every print in the library
goes through nltk.termsec.safe_print, which renders control
characters, escape sequences and unbalanced bidi overrides from
corpus text as visible \xNN escapes instead of letting them reach
the terminal. The CSV files the twitter and sentiment helpers
write neutralise formula-leading and control-sequence cells,
AlignedSent DOT labels are escaped, the WordNet browser escapes
corpus text, and nltk.data.load's verbose output no longer prints
the full resource URL (PR #3914, #3915, #3937, #3946, #3931,
#3930, #3856, #3902)
- Corpus stores and encodings: the PanLex Lite and chat80 sqlite
stores open read-only on a pinned descriptor with their schema
screened and a statement step budget, chat80 shelves are read by
a key manifest, Toolbox SFM markers are sanitised, and a resource
encoding that is not a text codec (bz2_codec and friends) is
refused (PR #3961, #3964, #3959, #3888, #3800, #3962)
- Package contents: the wheel no longer ships nltk.test.unit (the
attack harnesses, advisory probes and their fixtures), so pip
install nltk is about half the size and import nltk.test.unit
works from a source checkout or the sdist only. The doctests
under nltk.test still ship (PR #3954)
Security test and CI infrastructure
- Attack and regression harnesses, advisory probes, the sandbox
conftest fixtures, the pre-commit guards (no unsandboxed open,
every regex through redos, every pickle through picklesec, every
print through safe_print), load-invariant CPU-time timing rules,
CI matrix and runner fixes, pinned third-party downloads, and the
AUTHORS update (PR #3744, #3749, #3762, #3764, #3770, #3776,
#3791, #3795, #3803, #3818, #3820, #3821, #3822, #3849, #3855,
#3860, #3861, #3862, #3864, #3865, #3866, #3876, #3884, #3899,
#3904, #3905, #3912, #3917, #3924, #3926, #3927, #3939, #3940,
#3943, #3948, #3949, #3963, #3968, #3971, #3976, #3982, #3983,
#3984, #3985)
Other changes
- Added RTL language syntactic tree visualization support (PR
#3472)
- Fix#3394: Ensure punkt_tab is downloaded when punkt is requested
(PR #3505)
- Show README.md as the PyPI long description (PR #3746)
- Add stopwords.words("english-extended") (PR #3747)
- Krippendorff's alpha: treat placeholder labels as missing data
(#2865, #2732) (PR #3752)
- Fix shared defaults in chat80 Concept (PR #3773)
- Fix Java wrappers decoding str output from java()
(universal_newlines) (PR #3777)
- Prevent IndexError in Hungarian stemmer on empty string (PR
#3785)
- Isolate MTE tag filters per view (PR #3792)
- Fix newline in ellipsis tokens in TweetTokenizer (PR #3819)
- Isolate default WordNet reference relations (PR #3872)
- Fix resolution paths that cancel distinct literals (PR #3878)
- Fix UTF-8 text preprocessing in NKJP corpus reader (PR #3879)
- Remove shadowed FeatStruct._walk stub and preserve its docstring
(PR #3893)
- Count all increasing pairs in RIBES kendall_tau (PR #3900)
- Fix the TweetTokenizer ZWJ emoji pattern capturing adjacent
whitespace (PR #3913)
- Surface the CoreNLP server's reason for a non-2xx answer (PR
#3942)
- Omit unaligned (NULL target) pairs from Alignment strings, so IBM
model output prints (PR #3967)
- Fix DependencyGraph cycle detection, which returned False on
every parsed graph (PR #3978)
- Show, range and draw an alignment with an unaligned pair on every
surface (PR #3987)
What's Changed:
- Show README.md as the PyPI long description
- Add stopwords.words("english-extended")
- Fix pathsec.urlopen treating NO_PROXY as a configured proxy
(#3748)
- Krippendorff's alpha: treat placeholder labels as missing data
(#2865, #2732)
- Harden core security modules: pathsec URL-scheme bypass,
picklesec denylist, xmlsec parse()
- Add a living per-advisory security-probe harness
- Harden tokenize file I/O against path traversal (punkt,
stanford_segmenter)
- Add a shared pathsec sandbox conftest fixture
- Contain CorpusReader / agreement / chat80 file I/O in the pathsec
sandbox
- Write parse demo/profile output to a private staging dir, not
/tmp or the CWD
- Add pathsec regression tests for sem.util / xmlsec / dataset
loaders
- Harden maxent / named-entity model-artifact saves against
shared-tmp planting
- Route weka ARFF_Formatter.write through the pathsec sandbox
- Harden untrusted-model pickle allowlist (exact globals +
numpy.lib denylist)
- Split nltk_data pickle-asset regression into always-on and
skippable tests
- Reject Java launcher switches and @argfiles in the java() cmd
argument (CWE-88)
- Prevent infinite loop from cyclic collection indices
- Fix ARFF label injection in Weka classifier
- Stabilize multiprocessing test timeouts and context handling
- Fix Java wrappers decoding str output from java()
(universal_newlines)
- Fix shared defaults in chat80 Concept
- Harden internals.java() (CWE-88): options, classpath, env,
discovery; route MaltParser through it
- Bound decompression entry points in nltk (zip/gzip layers +
pathsec.ZipFile + weka), CWE-409
- Restore pre-commit formatting and register the new probe id on
develop
- Harden pickle deserialization (picklesec) against
arbitrary-code-execution gadgets
- Fix internals.doctest to match the hardened
_validate_java_options allowlist
- Bound re_show's caller-supplied regex against ReDoS (CWE-1333)
- Refuse CWD-relative matches for bare names in find_file/find_dir
(CWE-427)
- Bound RegexpStemmer's caller-supplied pattern against ReDoS
- Deserialize chat80 shelve values through the restricted unpickler
(RCE fix)
- Harden path/resource/tool validation in nltk.pathsec
- Harden downloader Package attributes and extraction against
traversal/Zip-Slip
- Harden nltk.xmlsec.parse: read filename sources through
pathsec.open
- Validate the NE chunker save_params destination through the
tool-directory guard
- Pin maxent scratch files to a data root and validate scratch IO
- Add staging_tempdir and tighten data.py zip/staging guards
- Stage boxer scratch input inside an nltk.data root
- Canonicalize obfuscated numeric IPv4 in the SSRF guard (CWE-918)
- Harden the VERSION read in nltk package init with
pathsec.open_package_resource
- Harden stanford tokenizer/segmenter wrappers: validate jar/model
paths and argv
- Harden tagger model loads and tool paths
(perceptron/crf/hunpos/stanford)
- Stage weka scratch files in a data root and validate the WEKAHOME
jar path
- Route corpus-reader scratch tempfiles through a data root and
label framenet path refusals
- Harden malt and stanford parser wrappers: validate
binary/model/IO paths and argv
- Harden nltk.tbl.demo model load/save through picklesec and
pathsec
- Add tests covering already-merged pathsec/security guards
- Fix develop CI: register data roots for benign save/IO tests
broken by pathsec hardening
- Prevent combinatorial DoS in skipgrams
- Prevent IndexError in Hungarian stemmer on empty string
- Sanitize SFM markers in ToolboxData
- Isolate MTE tag filters per view
- Prevent classpath shadowing in GenericStanfordParser
- Prevent batch-boundary injection in StanfordParser
- Add remaining GHSA-8mgp security regression tests
(pathsec/java/redos/archive/corpus)
- Add redos.source_of/reharden and picklesec.harden_object_graph
- Load nltk.tbl.demo models through an allowlisting unpickler
- Bound algorithmic-complexity (time) DoS in stemmers, metrics,
parsers (CWE-407 / CWE-770)
- Bound regex compilation against a compile-time DoS
- Route top-level nltk regexes through nltk.redos
(CWE-1333/CWE-400)
- Route nltk.sentiment regexes through nltk.redos
(CWE-1333/CWE-400)
- Route nltk.chat regexes through nltk.redos (CWE-1333/CWE-400)
- Route nltk.corpus regexes through nltk.redos (CWE-1333/CWE-400)
- Route nltk.parse regexes through nltk.redos (CWE-1333/CWE-400)
- Route nltk.draw regexes through nltk.redos (CWE-1333/CWE-400)
- Route nltk.stem regexes through nltk.redos (CWE-1333/CWE-400)
- Route nltk.translate regexes through nltk.redos
(CWE-1333/CWE-400)
- Route nltk.sem regexes through nltk.redos (CWE-1333/CWE-400)
- Route nltk.tree regexes through nltk.redos (CWE-1333/CWE-400)
- Route nltk.tokenize regexes through nltk.redos (CWE-1333/CWE-400)
- Route nltk.chunk regexes through nltk.redos (CWE-1333/CWE-400)
- Route nltk.classify regexes through nltk.redos (CWE-1333/CWE-400)
- Route nltk.tag regexes through nltk.redos (CWE-1333/CWE-400)
- Route nltk.app regexes through nltk.redos (CWE-1333/CWE-400)
- Route nltk.ccg regexes through nltk.redos (CWE-1333/CWE-400)
- Bound compile-time DoS (ValueError) in the concordance and nemo
GUI regex handlers (CWE-1333/CWE-400)
- Harden RegexpStemmer against pre-compiled ReDoS patterns
(CWE-1333)
- Add security regression tests + harden SSRF filter (transition
tunnels, IP literals)
- Strengthen the advisory security-probe harness (guards already on
develop)
- Guard model-artifact loaders against un-sandboxed open()
- Route security-test fixtures through the pathsec I/O helpers
- Stage REPP scratch input in an nltk.data root instead of the
shared system temp
- Bound JSON deserialization against deep-nesting crash and
resource DoS
- Fix newline in ellipsis tokens in TweetTokenizer
- Refuse NUL bytes in pathsec.validate_path; add a pathsec
attack-candidate harness
- Re-check the normalized form of a no-protocol resource path
against traversal (CWE-22)
- Add and expand security / ReDoS attack test coverage
- Escape corpus-derived text in the WordNet browser app (CWE-79
XSS)
- Add test-only regression coverage for already-fixed CWEs
- Extend the no-unsandboxed-open guard to app/draw/twitter and
route every open through pathsec.open
- Add the all-regex-through-redos pre-commit guard and two
regression tests
- Exclude pytest's ephemeral basetemp from the pickle-asset scanner
- Open operator-chosen app/twitter paths with an annotated builtin
open, not pathsec.open
- Bound the WekaClassifier model path to the pathsec data roots
- Tidy transitionparser security comments to one point per <=3
lines
- Deflake anti-DoS timing tests and harden senseval _fixXML
leading-run ReDoS
- Harden and standardize NLTK's external-tool wrappers on a pathsec
trusted-spawn API (CWE-426/427/732/93/88)
- Don't miscount a python-crfsuite Windows failure as a pathsec
over-block (tag sweep)
- Give every pathsec refusal the standard Security Violation marker
(spawn_trusted + ZIP audit)
- Bound recursion depth in parsers and traversal helpers
- Isolate default WordNet reference relations
- Harden and complete the CoreNLP server-option allowlist
(CWE-88/22/502)
- Optimize CI test matrix
- Harden CoreNLPServer start attempts
- Make DoS/ReDoS timing regression tests load-invariant (fix CI
flakiness)
- Extend no-unsandboxed-open guard to
gzip/bz2/lzma/zipfile/tempfile (+ coverage test)
- Count all increasing pairs in RIBES kendall_tau
- Potential fix for code scanning alert no. 28: Clear-text logging
of sensitive information
- Remove shadowed FeatStruct._walk stub and preserve its docstring
- Fix resolution paths that cancel distinct literals
- Add GHSA-j456 live probe; expand the GHSA-8mgp probe attack set
- Handle the redos MAX_PATTERN_LENGTH refusal in
help._format_tagset
- Handle the redos MAX_PATTERN_LENGTH refusal in
TokenSearcher.findall
- Handle the redos MAX_PATTERN_LENGTH refusal for a tgrep /regex/
node literal
- Make help._format_tagset fail closed on an oversized tag pattern
(fix develop CI)
- Add nltk.termsec terminal-output sanitizer with attack-matrix
tests
- [WIP] Fix quadratic DoS in Tree.chomsky_normal_form right-factor
binarization (GHSA-r53h-rw34-8h97)
- Bound SyllableTokenizer fallback-vowel accumulation (CWE-400 /
CWE-407)
- Fix develop CI on Python 3.10/3.11: let deepcopy reach
MAX_TREE_DEPTH
- Fix the XMLCorpusView ReDoS-test Windows CI flake (in-process op
timing + Windows autoload)
- Route every library print through the nltk.termsec.safe_print
chokepoint
- Log the perceptron training progress on the module logger (fixes
the develop 3.10/3.11 cells)
- Resolve every executed tool binary absolute-only: prover9/mace4,
megam, tadm, java, hunpos (untrusted search path, CWE-426/427)
- Pin the GHSA-cc5r-64rf-75hg proof of concept end to end and run
its sink as probe phase 6 (follow-up to #3887)
- Validate chat80.sql_query dbname for path containment (CWE-59/73)
- Extend the tool model-file guards to crf, stanford tagger and
segmenter
- Consolidate the internals, downloader, pathsec and decorators
guards from #3753 and #3850
- Consolidate the nltk/parse guards from #3753 and #3850 and judge
their inputs on real characters
- Sanitise the CSV cells and report lines the sentiment helpers
write
- Read a long line in linear time in
SeekableUnicodeStreamReader.readline (GHSA-j8g8)
- Escape AlignedSent DOT labels, refuse NUL, and keep the alignment
checks under -O
- Align quote tokens in linear time in the Treebank and NLTK word
tokenizers
- Never leave the pop of an escaped bidi override live in
sanitize_terminal
- Find readline's line boundary with str.splitlines, not a timed
regex
- Walk semi-relations once in relextract, and keep a pattern's
flags in redos.reharden
- Fix UTF-8 text preprocessing in NKJP corpus reader
- Make the CoNLL SRL, TIMIT and XML view readers linear, and parse
corpus XML through xmlsec (GHSA-v8f3)
- Measure probe scaling in process CPU time so a loaded runner
cannot flip a verdict
- ci(deps): bump dorny/paths-filter from 3 to 4 in the
github-actions group[bot] in
https://github.com/nltk/nltk/pull/3943
- Surface the CoreNLP server's reason for a non-2xx answer and warm
the live test server
- Guard the Stanford tagger and segmenter input against line
injection, verified on the real tools
- Bring the self-contained tests of #3753 to develop
- Guard the twitter CSV exports against formula and
control-sequence injection, verified on the real corpus
- Guard the maxent parameter files against outside-root paths and
unrepresentable names, verified on a real trained model
- Parse CCG lexicon categories in linear time and refuse oversized
ones, verified under attack and on the documented grammars
(GHSA-89p3-fcch-88ph, split from #3753)
- [Do not merge yet] Guard tool wrappers, Prover9 input and the
downloader against injection and unbounded-response DoS
- Added RTL language syntactic tree visualization support
- Make the word, blankline and TextTiling tokenizers, the valuation
parser, relextract and toktok linear on retried leading regex
runs
- Add the 3.10.4 commit authors and security reporters to
AUTHORS.md
- Pair the scaling samples by rep and size the under-margined
probes over the floor
- Trusted spawn: O_NOFOLLOW and inode check on the final file, link
owner and control-character checks, refusal reasons, argv[0] as
invoked; real Graphviz in CI (#3887 follow-up)
- Support symlinked executables in pathsec
- [Do not merge yet] Keep downloaded archives private and serve
other accounts through extraction (fixes#3928)
- Bound algorithmic-DoS and uncontrolled-recursion vectors
(CWE-400/407/674/1333)
- Guard the two-class any-character runs and count apostrophe-named
groups (follow-up to #3895)
- Fix#3394: Ensure punkt_tab is downloaded when punkt is requested
- Restore add_blank_lines and to_settings_string after the
GHSA-f2h2 depth bound
- Let chat80.val_load read back what val_dump wrote
- Keep full collections out of timed scaling samples (89p3 teeth
and relextract reds)
- CI: take the Windows Graphviz from the cached third-party tools,
not Chocolatey
- Close the chat80 store swap window and read stores by a key
manifest
- Route every module's file removal, rename, privacy test and
sqlite open through pathsec
- Fix the develop CI breakage from the fg9w advisory merge (Senna
-path separator, test fixtures under /tmp)
- Refuse relation walks deeper than MAX_RECURSION_DEPTH instead of
truncating them
- Harden the Toolbox reader and writers after the #3958 re-audit
- Route the downloader, XML corpus view, internals and Toolbox
through nltk.xmlsec and nltk.data
- Open gzip file names through pathsec, and refuse a linked PanLex
Lite database
- Fix DependencyGraph cycle detection, dead on every parsed graph
- Bound YAML deserialization against depth and resource DoS,
verified on the real loaders
- Guard java() cmd[0] against JEP 330 source-file execution,
verified on a real JDK
- Make Tree.__repr__ iterative so a deep accepted tree does not
overflow the stack
- Create perceptron model files exclusively under the pinned
directory (issue 3980)
- Drop the dash-banner section dividers from the model-artifact
tests
- Skip the real-launcher test when java resolves to a runtime-less
stub
- Pin the CI megam download to https and its sha256
- Run the advisory probes only on a full-matrix run; keep the unit
tests out of the wheel
- Charge the shipped-form teeth leg on CPU time (Windows regex
timeout is wall-clock)
- Prepare NLTK release 3.10.4
- Rewrite the 3.10.4 release notes around the user-facing security
changes
- Omit NULL targets from alignment strings
- Fix TweetTokenizer ZWJ pattern capturing whitespace
- Show, range and draw an alignment with an unaligned pair on every
surface (follow-up to #3967, #1885)
- Pin the TweetTokenizer ZWJ alternative: linear on repeated
shapes, no whitespace absorbed (follow-up to #3913)
- Credit every contributor since 3.10.3 in the 3.10.4 notes and
AUTHORS.md; name the PRs merged after #3986
- Make the span_tokenize teeth control compute-bound so Windows
reads its quadratic (deflake, test-only)
Signed-off-by: Leon Anavi <leon.anavi@konsulko.com>
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
Upgrade to release 9.1.0:
- Removed support for Python 3.10, now end-of-life.
- Where a path supplies more than one metadata directory for a
distribution, as an interrupted or unclean upgrade may leave
behind, discovery now prefers the greater one by reverse
lexicographic ordering of the directory names, so the stale
metadata no longer shadows the version in use.
- A missing required metadata field (Name or Version) now raises a
KeyError that names the field and identifies the distribution,
which PathDistribution renders as its metadata path.
- Restored lazy validation of EntryPoint.value, deferring the
ValueError for an invalid value until a component is accessed or
the entry point is loaded, so one malformed entry point no longer
prevents entry points in other groups from being selected.
- importlib_metadata.diagnose no longer crashes on a metadata
directory that has no metadata file or whose metadata declares no
name. Such distributions are now reported by their location.
Signed-off-by: Leon Anavi <leon.anavi@konsulko.com>
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
Upgrade to release 2.5.0:
- imgtool now requires Python 3.8 or later. This also fixes key
export commands, which failed on Python 3.8 and 3.9.
- Added imgtool keyinfo, which reports whether a PEM file holds a
key pair or only a public key. With --require, it fails if the
kind does not match, which build systems can use as a check.
- Added --name-suffix to imgtool getpub and imgtool getpubhash,
which appends a suffix to the generated C or Rust symbol names so
that several keys can be embedded in one image.
- Public-key-only PEM files are now supported by imgtool getpub,
imgtool getpubhash and imgtool verify. imgtool sign given a
public-only PEM now fails with a clear error.
Signed-off-by: Leon Anavi <leon.anavi@konsulko.com>
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
Upgrade to release 2.16.0:
Changed:
- Look up only the cookie domains that match the request host, and
skip the expired cookie sweep until a cookie is due. With 1,000
cookies across 500 domains, building the Cookie header is about
18 times faster. Cookies from different domains are now sent from
the least to the most specific domain.
Fixed:
- Reuse the client cookie jar instead of copying it on every
request and redirect. A custom policy set with
client.cookies.jar.set_policy() now applies to outgoing requests
Signed-off-by: Leon Anavi <leon.anavi@konsulko.com>
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
Upgrade to release 7.17.0:
- Dropped support for Python 3.10.
- Binary wheels are now provided for iOS and Android. This allows
the use of the optimized C tracer on mobile platforms.
- Fix: three or more levels of nested with statements could make a
with line look like a branch with an exit that can never happen.
Using the sys.monitoring core, these phantom branches were
reported as partial. Other cores hid the problem but still
counted the phantom branches in the total.
- Fix: a comment or string that merely mentioned if True: or while
True: could mark its line as a branch that is known to be
partial, so a branch that never ran was reported as taken. These
built-in patterns now only match at the start of the statement.
- Fix: region detection missed functions defined in unusual places,
such as else and except clauses. This is now fixed.
- Fix: recent versions of the lcov tool complain if the same
function name appears more than once in a report. Python can have
multiple same-named functions in a few ways, for example,
getters/setters or conditionally defined functions. These are now
disambiguated by adding a @<line number> suffix to the function
name.
- Fix: The "Can't combine statement coverage data with branch data"
error now names the two data files it couldn’t combine. The error
description in the docs now lists common causes and how to find
the process that wrote the file.
- Fix: at coverage totals close to 0 or 100, the XML report could
get incorrect line-rate values like 8.333e-05 or 1. This is now
fixed.
- Fix: a parallel data file saved more than once kept the hash of
its first save in its file name. Combining could then skip it as
a duplicate of another file that had the same data at that point,
and delete it, losing the data recorded after the first save.
Each save now updates the hash. This had been a problem since
7.13.1.
- Fixes for obscure but legal file names:
- File names (and plugin names) can contain control characters,
which are legal in POSIX paths but unsafe in reports: in XML
1.0 they make the document non-well-formed, and in a terminal
they act as escape sequences (colors, title/clipboard
sequences, cursor moves that forge output). Now the XML report,
the text and Markdown summary reports, and coverage debug data
replace non-printable characters in names. Printable names are
unchanged.
- A file pattern (from include, omit, or a [paths] alias)
containing a newline sent the glob-to-regex translation into an
infinite loop, hanging the process. Newlines in patterns are
now handled like any other character.
- Fixes in the implementation of the C tracer:
- If a plugin’s dynamic_source_filename method raised an
exception, lines in that function could be recorded as
belonging to a different file. Also fixed some minor memory
leaks in the C tracer.
- The refcount of None was mismanaged during function calls,
including comprehensions. This was unlikely to cause an actual
problem, but is now fixed. Thanks, Andrew Dalke.
- The C tracer held borrowed references to dictionary values,
which was unsafe on free-threaded Python. Now it takes its own
references.
- An internal error while recording a new file could be reported
as a confusing SystemError instead of the actual exception.
- Performance: the text summary report (coverage report) held on to
every file’s parsed source until it finished, so its memory use
grew with the size of the project: ten times or more what 7.14
used on large projects. Now it keeps only what it needs to print.
- Performance: HTML reports should be about 5% faster to produce.
- Kept the political banner in the README.
Signed-off-by: Leon Anavi <leon.anavi@konsulko.com>
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
Upgrade to release 26.10.2:
- Fix reading the TLS peer certificate
(TransportDetails.peer_cert) with pyOpenSSL 26.2.0 and later:
autobahn.twisted.util.extract_peer_certificate() raised
AttributeError: 'X509' object has no attribute 'get_extension'
for every certificate with extensions, as pyOpenSSL removed its
X.509 extension API (and deprecates the rest of its X509 API).
The certificate is now read with cryptography, in the new
backend-neutral autobahn.util.peer_certificate_details(), which
also never raises on certificate content it cannot parse (it is
peer data, read before authentication): extensions, or a subject
or issuer, that cannot be read are left empty. The peer_cert dict
keeps its keys and formats, except: ⚠ extensions[].value was
OpenSSL's text; a subject or issuer alternative name is still
given as OpenSSL prints it (DNS:localhost, IP Address:127.0.0.1),
other extensions now as cryptography prints them; and
signature_algorithm is ed25519 instead of ED25519 for Ed25519
certificates
- Fix the TLS peer certificate never being available on Twisted
WebSocket and RawSocket connections: it was read when the TCP
connection was made, before the TLS handshake, and so was always
None. It is now read when the TLS handshake is complete, together
with the TLS channel ID. With this, TLS client certificate
authentication (Crossbar.io authmethod tls, and dynamic
authenticators reading transport.peer_cert) works again
- Fix asyncio: fill TransportDetails.peer_cert (it was always
None), and treat a TLS server connection without a client
certificate as secure, with a TLS channel ID: is_secure was
derived from the peer certificate, so WAMP-cryptosign with
channel binding failed on asyncio servers
- Behaviour change - the TLS channel ID tls-unique on Twisted now
follows RFC 5929 (the first TLS Finished message of the
handshake, in a full handshake the client's), as Python's ssl
module, and so autobahn on asyncio, already did. Twisted used the
server's Finished message, so WAMP-cryptosign with channel
binding failed between Twisted and asyncio peers over WebSocket,
and between Twisted RawSocket peers (whose protocols did not set
is_server, so the client side used the wrong message). Twisted
peers on autobahn 26.10.1 or older and on 26.10.2 no longer agree
on the channel ID over WebSocket: upgrade both sides of
connections that use channel binding (#2014) Work around a PyPy
bug: PyPy's ssl module (up to at least PyPy 7.3.23) returns the
server's TLS Finished message from
get_channel_binding("tls-unique") instead of the first (the
client's), so autobahn on asyncio on PyPy disagreed with all
other peers on the channel ID. The new autobahn.util.tls_unique()
uses PyPy's own tls_unique_cb() on PyPy, which returns the right
message
- TransportDetails.is_server is now set on Twisted RawSocket and
asyncio WebSocket connections (it was None)
Signed-off-by: Leon Anavi <leon.anavi@konsulko.com>
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
Upgrade to release 1.20.1:
bug:
- [bug] [autogenerate] Fixed issue where a table or column comment
set to an empty string would cause autogenerate to detect a
comment change on every run. As most backends, including
PostgreSQL, MySQL, MariaDB and Oracle, don't store an empty
string comment distinctly from no comment, autogenerate now
treats an empty string comment the same as None when comparing
comments. Additionally, an empty string comment is now rendered
in op.create_table() and op.add_column() directives, where it was
previously omitted. Pull request courtesy vineoy.
- [bug] [autogenerate] [postgresql] Fixed rendering of the
PostgreSQL ~sqlalchemy.dialects.postgresql.DOMAIN type in
autogenerated migration scripts. Previously the type was
rendered using repr(), which omitted the check, not_null,
default, collation, constraint_name, schema and create_type
arguments and did not apply a dialect prefix to the underlying
data type. The full set of arguments is now rendered and the
underlying data type is rendered using the standard
type-rendering logic. Pull request courtesy Sebastian Cao.
- [bug] [autogenerate] [postgresql] Fixed false-positive "Computed
default cannot be modified" warning emitted during autogenerate
when comparing PostgreSQL Computed columns whose reflected SQL
expression includes type cast specifiers such as ::regconfig or
::text, which PostgreSQL adds to the expression as stored. These
cast specifiers are now stripped before comparison so that
otherwise equivalent expressions no longer trigger the warning.
Pull request courtesy Sebastian Cao.
- [bug] [autogenerate] [postgresql] Fixed issue where autogenerate
would detect an unchanged PostgreSQL expression-based index as
modified when the expression included type casts, such as a cast
to a type with a length or precision, a nested cast, or a cast
followed by a DESC modifier.
- [bug] [postgresql] Fixed bug where Operations.alter_column()
against a PostgreSQL identity column would render an incomplete,
invalid ALTER TABLE ... ALTER COLUMN statement with no clause at
all when existing_server_default was omitted, even if there were
no existing server default. Pull request courtesy Riti Grover.
- [bug] [environment] Fixed issue where a third party plugin
published using the alembic.plugins entrypoint, configured to
refer to a plugin module as documented at
alembic.plugins.toplevel, would cause import alembic to fail with
a TypeError, as the loaded module was erroneously iterated rather
than being set up directly. An entrypoint that refers to a
collection of modules, the form which inadvertently worked
previously, continues to be accepted but now emits a deprecation
warning.
- [bug] [autogenerate] Fixed issue where dialect-specific keyword
arguments on a ForeignKeyConstraint, ForeignKey or
CheckConstraint, such as postgresql_not_valid, were not rendered
for the constraints that are part of an autogenerated
Operations.create_table() directive, nor for an autogenerated
Operations.create_check_constraint() directive. The
Operations.create_table() case additionally requires SQLAlchemy
2.0.55 / 2.1.5 or greater.
- [bug] [autogenerate] Fixed issue where, when using SQLAlchemy
2.1, the CHECK constraint generated by a type such as
~sqlalchemy.types.Enum or ~sqlalchemy.types.Boolean with
create_constraint=True would be rendered by autogenerate as
explicit sa.CheckConstraint() entries within op.create_table(),
in addition to the type itself which already produces that
constraint, leading to a duplicate constraint error when the
migration is run.
- [bug] [operations] Fixed issue where a CreateTableOp produced
from an existing ~sqlalchemy.schema.Table using
CreateTableOp.from_table() would emit the CHECK constraint
generated by a type such as ~sqlalchemy.types.Enum or
~sqlalchemy.types.Boolean with create_constraint=True twice in
the resulting CREATE TABLE statement when invoked directly.
Signed-off-by: Leon Anavi <leon.anavi@konsulko.com>
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
Drop the Cython version patch, as upstream has a wider range now.
async-timeout was removed as an dependency upstream in v1.90.0.
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
Upgrade to addresses eMudhra certificates issue.
See https://emudhra.we-have-a-problem.com/
Sync all grpc recipes to the same version.
Removed obsolete patches:
* The compiler-argument fix is already handled upstream using
shlex.split()
* Bundled protobuf now restricts musttail to AArch64 and x86-64, so the
MIPS exclusion patch is unnecessary.
License-Update: switched from license to license expression
orig: License: Apache License 2.0
New: License-Expression: Apache-2.0
Signed-off-by: Peter Marko <peter.marko@siemens.com>
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
std only provides AtomicU64 on targets with 64-bit atomics, so ruff
fails to build for riscv32:
error[E0432]: unresolved import `std::sync::atomic::AtomicU64`
--> crates/ruff/src/cache.rs:7:25
Add a patch that uses portable-atomic's AtomicU64 for the cache
timestamp on such targets, archived in the same on-disk format.
AI-Generated: Uses Claude Code
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
docutils 0.23 serializes boolean attributes as "1"/"0" rather than
"True"/"False", which made almost every test_examples case fail; add a
patch that normalizes the value when comparing.
The class, cpp_function, group and headings examples compare against
models generated with doxygen 1.9.x and do not match doxygen 1.18
output, so deselect them in a custom run-ptest.
AI-Generated: Uses Claude Code
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
fsspec imports ctypes, asyncio, json, logging and threading, which are
not in python3-core, without ctypes "import fsspec" fails, which also
broke the python3-pyyaml-include ptests.
Install the top-level fsspec/conftest.py into the ptest dir, it
provides the "m" fixture and the autouse instance cache cleaner, and
without it most tests error out or fail.
Signed-off-by: Khem Raj <khem.raj@oss.qualcomm.com>
Leon Anavi
Khem Raj
Ankur Tyagi
Ross Burton
Khem Raj
Peter Marko