Changelog
The following page lists all (technical) changes in the extraction plugin SDK.
Programming language specific API changes are described in more detail on API changelog pages. These pages list new API functionalities, and describe how to update your plugins when API changes are in order. For the API changelog pages see:
Release-0.8.3
HANSKEN-21097: Keep health and pluginInfo calls responsive when plugin is busy (Python).
HANSKEN-21132: Fail with a clear message when reading data of a trace without data (Python).
Release-0.8.2
HANSKEN-21104: Reduce the level of details for some debug-logs (Python)
HANSKEN-21130: Use a
default_buffer_size
of 6M for trace.open() calls.HANSKEN-21096: Support new
buffer_size
argument also for searched traces data reads (.open(buffer_size=...)
).HANSKEN-20962: Add option to specify read and write buffers, and support mode
w
fortrace.open(...)
(python).HANSKEN-20938: Update
build_plugin
, to not require a fully installed environment when running building and labeling a plugin.HANSKEN-20852: Plugin client: change log level of wrapped StatusRuntimeExceptions from INFO to DEBUG for less verbose logging.
Release-0.8.1
HANSKEN-20824: Performance improvement Python: never execute reads of size 0 over grpc (e.g. when using pillow).
HANSKEN-20812: Relax python sdk dependencies (to be compatible with Tensorflow).
HANSKEN-20814: Support lists of ints and doubles in the test framework
HANSKEN-20813: Let test framework write runtime profiles to txt files in tests results folders
HANSKEN-20735: Add basic profiling to Python extraction plugins for performance debugging in Hansken.
HANSKEN-20733: Set number of gRPC connections equal to the number of workers, to support improved better load balancing strategies (Python).
Release-0.8.0
HANSKEN-20128: Scope trace searches by default to the image under extraction, but allow project-wide searches by passing an optional argument (Java).
HANSKEN-20127: Scope trace searches by default to the image under extraction, but allow project-wide searches by passing an optional argument (Python).
HANSKEN-20552: Support trace property types of list[float] (Python), List
(Java), List (Java), and List (Java). HANSKEN-20389: Update project dependencies
Release-0.7.4
HANSKEN-20410: Fix
label_plugin
andbuild_plugin
were not packaged correctly
Release-0.7.3
HANSKEN-19436: Introduced
label_plugin
utility as replacement forbuild_plugin
Release-0.7.2
HANSKEN-19954: Wait for slow plugins to start before testing (Python test framework)
HANSKEN-14216: Use Docker API instead of subprocess (Python test framework)
Release-0.7.1
HANSKEN-19759: Fix bug where wildcard in matcher query did not match on multiline values
HANSKEN-19064: Allow plugins to import local modules (Python) (second attempt)
HANSKEN-19448: Prettier exception when
test_plugin
can not load pluginHANSKEN-19447: Test java and docker in test framework, to give the user better error messages if java/docker is missing or is outdated
HANSKEN-19435: Unclutter test framework output in case of failure and fail with correct exit code (Python)
HANSKEN-19151: Added configurable
workers
to the plugin resources
Release-0.7.0
HANSKEN-18677: Added HQL-Lite auto-escaping & fixed fullmatch wildcard support
HANSKEN-19179: Update all project dependencies
HANSKEN-19148: Improve build_plugin and test framework output (by passing subprocess output directly to the terminal)
HANSKEN-19076: Exposed API version through the ExtractionPluginClient
HANSKEN-18830: Added the Hansken AIO
DebugExtractionPluginTool
to the docsHANSKEN-19065: Update link to extraction plugin examples (now hosted on Github)
HANSKEN-19064: Allow plugins to import local modules (Python)
HANSKEN-17675: Enable writing streaming data from Python
HANSKEN-18982: Improve error messages for client-side exceptions caught by gRPC (also for deferred plugins)
Release-0.6.3
HANSKEN-18915: Update all project dependencies
HANSKEN-18907: Upgrade to antlr 4.9.3
HANSKEN-18883: Allow writing multiple datastreams to a trace concurrently
HANSKEN-18673: Improve error messages for client-side exceptions caught by gRPC
HANSKEN-18517: Improve Python documentation for creating nested traces
HANSKEN-18400: Building docs with tox uses docutils 0.18.1
HANSKEN-17556: Enables writing to traces/child traces out of order
Release-0.6.2
HANSKEN-17692: [EXPERIMENTAL] Add Python support for
previews
, and extend test-framework to support them as wellHANSKEN-17742: Raise FileNotFoundError instead of logging an error when the plugin’s file does not exist
HANSKEN-17786: Update grpc dependencies to 1.48.2 to fix CVE-2022-3509
HANSKEN-17636: Improve explanation on “Match on specific datastream type”
HANSKEN-17672: Add a nice JB cartoon to the SDK docs landing page
HANSKEN-17502: Fix org.hansken.plugin-info.id label for java plugins
HANSKEN-17460: Update Flits to 3.7.1
Release-0.6.1
HANSKEN-17265: Added a parameter to build_plugin.py to extend the docker command for proxy settings
HANSKEN-17264: Remove PortUtil and expose listening ports
HANSKEN-17276: Move external getting started to the SDK documentation
HANSKEN-17274: Fix broken external links to Hansken website
HANSKEN-17278: Add url and license to SDK Python package info
HANSKEN-17203: Publish new SDK releases to maven central (Java)
HANSKEN-17277: Rearrange code snippets documentation
HANSKEN-17273: Update Checkstyle
HANSKEN-17214: Fix issues from static code analysis
Release-0.6.0
HANSKEN-17196: Downgrade API compatability level to 0.5.0
HANSKEN-17194: Update project dependencies to latest versions
HANSKEN-16781: Remove the need for
plugin=self
and movedid
to 1st arg when creating a PluginInfo in Python SDKHANSKEN-17191: Add quicklinks to the documentation index
HANSKEN-16756: Read extraction plugin version from plugin mavens project.version
HANSKEN-16705: Improve apis for declaring plugin resources and querying the data context at runtime
HANSKEN-17151: Store Plugin info in plugin image labels for Python plugins
HANSKEN-16753: Store Plugin info in plugin image labels for Java plugins
HANSKEN-17178: Updated Python tracelet documentation to contain a working example
Release-0.5.1
HANSKEN-17141: Fixed
$data.type=...
matcher when usingrun_with_hanskenpy
(Python)HANSKEN-16908: Added gRPC health service for Python Extraction Plugins
HANSKEN-17138: Avoid multiple plugins running on the same hostname:port (Python)
HANSKEN-14755: Use Python test framework wrapper with manually started plugins
HANSKEN-16905: Added gRPC health service for Java Extraction Plugins
HANSKEN-16901: Exclude old guava version to be able to run RemoteExtractionPluginFlits from Intellij
HANSKEN-16900: TestTraceSearcher will now return searched traces in natural sorted order of their file names
HANSKEN-16725: Improve tracelet api in Python SDK
HANSKEN-16764: Corrected ‘Adding data to traces’ code snippets
HANSKEN-16704: Use Python’s dataclasses where applicable
HANSKEN-17064: Improve runtime packing type checks to allow Sequence / Mapping compatible values
Release-0.5.0
HANSKEN-16638: Improve tracelet api in Java SDK
HANSKEN-16576: Support vector data type in Python SDK
HANSKEN-16707: Update build dependencies, build and test for Python 3.10
HANSKEN-16575: Support vector data type in Java SDK
HANSKEN-16574: Support vector data type in common SDK
Release-0.4.14
HANSKEN-16632: Fix execution of meta-extraction plugins with hansken.py runner which failed with an ‘expected $data in matcher’ error
HANSKEN-16634: Allow forward compatibility with Hansken by introducing an GRPC API version
HANSKEN-16489: Let build pipeline publish Java artifacts to community
HANSKEN-16558: Serve SDK test framework errors in a more developer-friendly way
HANSKEN-16489: Removed incompatibility warning for All In One with Hansken.py
HANSKEN-16403: Fixed running markdownlint with
tox -e markdownlint
HANSKEN-16268: Added
*
value support to the HQL-Lite term matcher, improved documentation by using HQL defaultproperty:value
instead ofproperty=value
HANSKEN-16258: Fixed Jenkins build
HANSKEN-16257: Fixed docker stop command in test framework
Release-0.4.13
HANSKEN-16229: Fixed SDK documentation not correctly zipped
HANSKEN-16095: Documentation: added Test framework data-stream type note
HANSKEN-15961: Redundant plugin errors in the client set to log level debug
HANSKEN-16111: Updated the testframework to skip search traces when scanning input files
HANSKEN-16128: Updated the testframework to disallow overwriting properties, similar to Hansken
HANSKEN-16191: Fixed Jenkins build
commit id could not be retrieved
curl didn’t work due to a bad proxy
HANSKEN-16116: Updated FLITS dependency to 3.5.2
HANSKEN-16105: Seeking beyond EOF for Python throws an exception
HANSKEN-16160: Updated remaining JUnit 4 tests to JUnit 5
HANSKEN-16118: Trace ids are now optional when writing automated tests using Flits
HANSKEN-16012: Fixed trace types were not correctly determined from property names (Python)
HANSKEN-16238: Changed default log level of extraction plugin server. Added command line option to increase it.
Release-0.4.12
HANSKEN-15857: Document HQL-lite for programmers manual
HANSKEN-16139: Run markdownlint in Jenkins instead of Docker
HANSKEN-16115: Updated log4j to version 2.16.0 due CVE-2021-44228
HANSKEN-15651: Added documentation on test files
HANSKEN-16001: Improved Python exceptions for better readability
HANSKEN-16044: Fixed documentation being unstashed to wrong directory on Jenkins
HANSKEN-14586: Documentation: added code snippets for adding a Datastream
HANSKEN-16038: Hansken.py: fix deferred tools
HANSKEN-15653: Added documentation on use with Hansken.py
HANSKEN-15801: Change build agent in Jenkins file
HANSKEN-16090: Fix for documentation build (m2r2 requires mistune < 2.0.0)
HANSKEN-15771: Document trace properties that mismatch the trace model
Release-0.4.11
HANSKEN-16048: Fix documentation was not released because the documentation zip was located in
docs/_build/docs
Release-0.4.10
HANSKEN-16037: Repair release
Release-0.4.9
HANSKEN-15858: Add a link to the Extraction Plugin SDK API Javadocs
HANSKEN-15656: Documentation: expanded FAQ
HANSKEN-15993: Added
isVerboseLoggingEnabled()
method toExtractionPluginFlits
to enable verbose logging if desiredHANSKEN-15801: Change build agent in Jenkins file.
HANSKEN-15766: Documentation: Mention the Getting Started guide on gitlab
HANSKEN-15765: Added documentation on the ‘Anatomy of a plugin’
HANSKEN-15772: Added documentation on tracelets
HANSKEN-15770: Added debug documentation (Java and Python)
HANSKEN-15773: Added documentation linter (markdownlint)
HANSKEN-15964: Added –verbose option to Python test_plugin runner
HANSKEN-15913: Use traceUid instead of traceId to read data
Release-0.4.8
HANSKEN-15668: Bugfix: prevent search from crashing plugin if results contain traces from different images
Release-0.4.7
HANSKEN-15871: Fix for RpcUnixTime & RpcZonedDateTime that were parsed incorrectly in Python
HANSKEN-15745: Document naming convention and added convienience method
id(domain, category, name)
to the PluginInfoBuilderHANSKEN-15790: Change dependency pinning policy
HANSKEN-15790: Fix typing issues discovered by upgrading mypy-protobuf
HANSKEN-15650: Added documentation on general concepts
HANSKEN-15743: Fix Test framework should not limit input files to 2G
HANSKEN-15846: Improved hansken.py matcher, instead of $data.type all $data matchers are supported
Release-0.4.6
HANSKEN-15711: Changed PluginResources cpu/memory values to floats
HANSKEN-15683: Changed the trace enrichment order to ensure transformations are handled before properties
HANSKEN-15589: Extend PluginInfo with the ability to specify plugin resources (Python)
HANSKEN-15588: Renamed PluginInfo pluginResources() to resources() (Java)
HANSKEN-15588: Extend PluginInfo with the ability to specify plugin resources (Java)
Release-0.4.5
deprecated release
Release-0.4.4
deprecated release
Release-0.4.3
HANSKEN-15641: Changed PluginId to use forward-slashes instead of backward-slashes
HANSKEN-15607: Explicitly fail deferred extraction plugins when building PluginInfo if the number of provided iterations is invalid
Release-0.4.2
HANSKEN-15632: Changed PluginInfo.license field to optional, for backwards compatibility
Release-0.4.1
deprecated release
Release-0.4.0
HANSKEN-15542: Extended Python PluginInfo with a license name and id consisting of domain, category and name
HANSKEN-15466: Extended Java PluginInfo with a license name and id consisting of domain, category and name
HANSKEN-15572: Include path as a property to trace type in TraceToJson
HANSKEN-15034: Add deferred tools to documentation
HANSKEN-15541: Publish SDK documentation as maven artifact
HANSKEN-15562: Fix test framework does not deserialize intrinsic properties
HANSKEN-15365: Create a FLITS test for deferred extraction plugin
Release-0.3.0
HANSKEN-15527: Add api changelogs for version 0.3.0
HANSKEN-15393: Rename Transformation to DataTransformation and RangedTransformation to RangedDataTransformation
HANSKEN-15391: Add support for ranged transformations (Java only)
HANSKEN-15390: Add proto definitions for ranged transformations
HANSKEN-15392: Add support for ranged transformations (Python)
Release-0.2.0
HANSKEN-15520: Add dedicated page for SDK API changes per language
HANSKEN-15515: Move Author and MaturityLevel to PluginInfo module (Python)
HANSKEN-15514: Refactor ExtractionContext to DataContext (Java and Python)
HANSKEN-15512: Move some internals from the EP python API to runtime module
HANSKEN-15511: Include API documentation in SDK dev docs
HANSKEN-15505: Cleanup SDK: remove unused position and unrequired datatype
HANSKEN-15491: Add compatibility check of remote plugin with current SDK version
HANSKEN-15502: Update TestRandomAccessData to accept dataType parameter
HANSKEN-15495: Make getData() lazy and replace getAllData() with getDataTypes()
HANSKEN-15498: Fixed isort configuration (Python)
HANSKEN-15029: Limit gRPC search request count
HANSKEN-15274: Add Python API for deferred extraction plugins
HANSKEN-15035: Allow python deferred plugins to run through hansken.py.
HANSKEN-15288: Add a new Trace subclass for search result Traces
HANSKEN-15042: Research and implement requesting data through GRPC for search traces
HANSKEN-15139: Allow deferred extraction plugins to process traces
HANSKEN-15015: Add option to create deferred extraction plugins using sdk
Release-0.1.8
HANSKEN-15236: Read initial chunk of data along with an RPC start request (Python)
HANSKEN-15338: Add setting of tracelets
HANSKEN-15370: Fixed ArrayOutOfBoundsException when there are bytes left when prefilling the cache
HANSKEN-15353: Added testframework exception result validation without or on partial message(startsWith, containsInOrder)
HANSKEN-15231: Added caffeine block cache implementation for Java RandomAccessData
HANSKEN-15276: Added support for Heterogeneous Maps
Release-0.1.7
HANSKEN-15282: Change the way reading data in the python sdk works. More BufferedReader functions are supported, including seeking
Release-0.1.6
HANSKEN-15294: Make Java SDK available to the Hansken community
HANSKEN-15233: Send initial chunk of data along with the start message (Java only)
HANSKEN-15232: Added RandomAccessData cache mechanism with fixed size of 1 MB to speed up large file reads
HANSKEN-15193: Improve client-side error message when server-side throws an exception
HANSKEN-15237: Added logging before and after processing a trace
HANSKEN-15187: Bugfix: Fixed flits traces with nested map properties were being parsed incorrectly
HANSKEN-15189: Added default log4j logging when no logging is configured
Release-0.1.5
HANSKEN-15192: Added Superpom for Java extraction plugins
HANSKEN-15186: Fixed matcher $data.mimeType does not work
Release-0.1.4
HANSKEN-14820: Bugfix: “Received a message from server, but processing of the trace has not been started yet”
HANSKEN-15059: Set up skeleton documentation for Extraction Plugin devs
HANSKEN-15048: Allow ‘workers’ as an optional argument when starting an ExtractionPluginServer
Release-0.1.3
HANSKEN-14787: Allow extraction plugins to be routed by HEADER by a proxy/loadbalancer
Release-0.1.2
HANSKEN-14923: Make SDK work on Windows
Release-0.1.1
HANSKEN-14867: Expanded HQL matcher to support Long & List (intrinsic)properties
Release-0.1.0
HANSKEN-14879: Allow SDK releases to be published on PyPI
HANSKEN-14738: Shade NFI internal projects into the SDK testframework jar
HANSKEN-14841: Bugfix where RpcStringMap wasn’t being unpacked properly in Python, which was discovered during a flits test.
HANSKEN-14703: Let Python plugins exit gracefully on SIGTERM
HANSKEN-14844: SDK: move serve from test_framework to runtime
HANSKEN-14793: Add ExtractionPluginBuilder.add_data method in python API
HANSKEN-14777: Add an extra check to ignore and log unsupported types during
RpcStart
gRPC serialization.HANSKEN-14739: License: Distribute
Extraction Plugin SDK
under the Apache License 2.0HANSKEN-14763: Bugfix where some python plugins were not loaded correctly when using serve or test-plugin commands
HANSKEN-14737: Move serve.py from plugin examples to SDK
HANSKEN-14720: Add option to use
with
when using thetrace.open
method in pythonHANSKEN-14582: Add option to write data using the python api
HANSKEN-14660: Move _test.py files from plugin examples repo to SDK repo
HANSKEN-14618: Add validation for unexpected extra data streams to test framework
HANSKEN-14704: Fix shading of the runtime super pom
HANSKEN-14619: Allow propagation of IOException in plugin new child callback
HANSKEN-14632: Add Java gRPC support for writing raw data streams on a trace
HANSKEN-14591: Split into three modules in the SDK
HANSKEN-14580: Add proto message definitions for raw data stream writing
HANSKEN-14635: Trace format which containes name/id can now be deserialized by testframework
HANSKEN-14131: Added ‘verbose’ logging for test-framework HQL matching
HANSKEN-14130: Updated StandaloneTestRunner to expose more errors & exceptions
HANSKEN-13784: Add meta support to test-framework
HANSKEN-14581: Extend Trace API with raw data writing capabilities
HANSKEN-14547: Deploy Java sources JAR for improved client debugging
HANSKEN-14531: Fix Python release. Python needs only one build step, which is either a snapshot or a release build. The separate Python release step was removed and merged with the first Python build step. Repository paths were corrected for the release version.
HANSKEN-14531: Fix python release - The python release is no longer a separate step in the build pipeline, since there is no actual difference between a snapshot and a release, apart from the version numbering scheme and the test-framework is downloaded from a repository location depending on the release parameter (see comments in Jenkinsfile)
HANSKEN-14161: Add Python test-framework wrapper around Java test-framework and add test-framework.tgz to whl
HANSKEN-14234: Restructure build pipeline to build and release Java first
HANSKEN-13799: Extraction Plugin: support meta extraction
HANSKEN-14286: Create adapter from RandomAccessData to InputStream
HANSKEN-14314: Don’t send child name when sending enrichment message
HANSKEN-14318: Flush cached children before flushing root in case of error with python gRPC server
HANSKEN-13414: Generate shaded jar for runtime that shades Guava, Protobuf, gRPC, and Netty (fix)
HANSKEN-14283: Allow passing a configuration of retry policy for the extraction plugin client
HANSKEN-14234: Restructure build pipeline to build and release Java first
HANSKEN-13414: Generate shaded jar for runtime that shades Guava, Protobuf, gRPC, and Netty
HANSKEN-14234: Restructure build pipeline to build and release Java first
HANSKEN-14135: Improve testing Python plugins in integration step and test reading large chunks
HANSKEN-14134: Fix releasing python plugins
HANSKEN-14128: Validate gRPC message limit for Python server instances
HANSKEN-14122: Fix missing comma in dependencies which broke the release
HANSKEN-14092: The type and total size of the data currently being processed can now be retrieved from an extractioncontext object passed to the process function
HANSKEN-13668: Added support for lists of longs, Hansken maps and LatLong to Java and Python API
HANSKEN-14104: Set Python gRPC limit to 64 MB(including message overhead)
HANSKEN-14079: Add logging to the SDK
HANSKEN-14010: Add support for serializing datetime in python API
HANSKEN-14083: Fix releasing python sdk
HANSKEN-14035: Add static type checks to python project
HANSKEN-14030: Allow test framework to be executed standalone for non-java extraction plugins
HANSKEN-14073: Support gRPC extraction plugins in test framework
HANSKEN-14074: Add support for serialization of Maps.
Hansken-14090: Use new Hansken python-api children call for creating nested children
HANSKEN-13774: Add support for creating children in python API
HANSKEN-13776: Add error handling to the Python based server and send error messages to the client
HANSKEN-14060: Remove mapping-interface from python extraction for API consistency
HANSKEN-13773: Update trace properties in Python API
HANSKEN-14044: Make sure python testing code is linted as well, enforce single quotes
HANSKEN-13772: Expose trace properties in Python API
HANSKEN-13775: Python - added trace.open() functionality to read from data streams
HANSKEN-14031: Split Trace interfaces (hansken.py trace vs external plugin trace)
HANSKEN-14037: Make sure pytest is always used for python tests
HANSKEN-14011: Implement unpack for trace
HANSKEN-14009: Implement pack for trace and trace enrichment
HANSKEN-14008: Move generated hql-lite parsers to different package (conflicts with hql package)
HANSKEN-13777: Added utility to run Python Extraction Plugin implementations with Hansken.py
HANSKEN-13771: Implement Extraction Plugin Info for Python plugins
HANSKEN-13966: Give socketproxy disconnect some time to disconnect (fixes flaky unit test)
HANSKEN-13810: Add extraction plugin python server code
HANSKEN-13676: Added support for ZonedDateTime over gRPC
HANSKEN-13922: Add webpage url to PluginInfo
HANSKEN-13676: Changed the way of creating child Traces to using a consumer.
HANSKEN-13655: Added server/client disconnect tests and implemented initial handling server-side
HANSKEN-13809: Added missing gRPC exception handles of process()
HANSKEN-13801: Made HQL-Lite matchers immutable
HANSKEN-13761: Seperated HQL-Lite type matcher implementation
HANSKEN-13756: Added HQL-Lite datastream matchers
HANSKEN-13800: (Temporarily) remove meta from Extraction Plugin API
HANSKEN-13798: Propagate exception on failure of START serialization
HANSKEN-13706: Create basic test framework implementation
HANSKEN-13769: Make jenkins run tests and a linter for python
HANSKEN-13705: Add support for creating child traces over gRPC
HANSKEN-13713: Make (partial) extraction plugin errors visible for clients
HANSKEN-13709: Send partial result when an external plugin errors out
HANSKEN-13733: Add script to generate gRPC Python files
HANSKEN-13660: Copied the Hql definition from Hansken to enable the HQL-Lite implementation
HANSKEN-13656: Test and handle invalid protocol messages
HANSKEN-13663: Add matcher interface to PluginInfo
HANSKEN-13714: Add IOException to ExtractionPlugin.process() interface
HANSKEN-13658: test(s) for non-grpc connection with a grpc server
HANSKEN-13650: Add basic support for writing trace information over gRPC
HANSKEN-13648: Add basic support for reading trace information over gRPC
HANSKEN-13651: Default implementations for RandomAccessData interface
HANSKEN-13643: Add basic support for reading trace information over gRPC
HANSKEN-13643: Add basic support for reading from trace data over gRPC
HANSKEN-13581: Plugin-info client/server implementations
HANSKEN-13580: Add gRPC server and client to serve an Extraction Plugin as a service
HANSKEN-13579: Created initial gRPC implementation messages
HANSKEN-13577: Be able to create releases of the SDK
HANSKEN-13578: Created plugin API
HANSKEN-13560: Create initial gRPC definitions
HANSKEN-13554: Initial repository