Changelog

The following page lists all (technical) changes in the extraction plugin SDK.

Programming language specific API changes are described in more detail on API changelog pages. These pages list new API functionalities, and describe how to update your plugins when API changes are in order. For the API changelog pages see:

Release-0.7.3

  • HANSKEN-19436: Introduced label_plugin utility as replacement for build_plugin

Release-0.7.2

  • HANSKEN-19954: Wait for slow plugins to start before testing (Python test framework)

  • HANSKEN-14216: Use Docker API instead of subprocess (Python test framework)

Release-0.7.1

  • HANSKEN-19759: Fix bug where wildcard in matcher query did not match on multiline values

  • HANSKEN-19064: Allow plugins to import local modules (Python) (second attempt)

  • HANSKEN-19448: Prettier exception when test_plugin can not load plugin

  • HANSKEN-19447: Test java and docker in test framework, to give the user better error messages if java/docker is missing or is outdated

  • HANSKEN-19435: Unclutter test framework output in case of failure and fail with correct exit code (Python)

  • HANSKEN-19151: Added configurable workers to the plugin resources

Release-0.7.0

  • HANSKEN-18677: Added HQL-Lite auto-escaping & fixed fullmatch wildcard support

  • HANSKEN-19179: Update all project dependencies

  • HANSKEN-19148: Improve build_plugin and test framework output (by passing subprocess output directly to the terminal)

  • HANSKEN-19076: Exposed API version through the ExtractionPluginClient

  • HANSKEN-18830: Added the Hansken AIO DebugExtractionPluginTool to the docs

  • HANSKEN-19065: Update link to extraction plugin examples (now hosted on Github)

  • HANSKEN-19064: Allow plugins to import local modules (Python)

  • HANSKEN-17675: Enable writing streaming data from Python

  • HANSKEN-18982: Improve error messages for client-side exceptions caught by gRPC (also for deferred plugins)

Release-0.6.3

  • HANSKEN-18915: Update all project dependencies

  • HANSKEN-18907: Upgrade to antlr 4.9.3

  • HANSKEN-18883: Allow writing multiple datastreams to a trace concurrently

  • HANSKEN-18673: Improve error messages for client-side exceptions caught by gRPC

  • HANSKEN-18517: Improve Python documentation for creating nested traces

  • HANSKEN-18400: Building docs with tox uses docutils 0.18.1

  • HANSKEN-17556: Enables writing to traces/child traces out of order

Release-0.6.2

  • HANSKEN-17692: [EXPERIMENTAL] Add Python support for previews, and extend test-framework to support them as well

  • HANSKEN-17742: Raise FileNotFoundError instead of logging an error when the plugin’s file does not exist

  • HANSKEN-17786: Update grpc dependencies to 1.48.2 to fix CVE-2022-3509

  • HANSKEN-17636: Improve explanation on “Match on specific datastream type”

  • HANSKEN-17672: Add a nice JB cartoon to the SDK docs landing page

  • HANSKEN-17502: Fix org.hansken.plugin-info.id label for java plugins

  • HANSKEN-17460: Update Flits to 3.7.1

Release-0.6.1

  • HANSKEN-17265: Added a parameter to build_plugin.py to extend the docker command for proxy settings

  • HANSKEN-17264: Remove PortUtil and expose listening ports

  • HANSKEN-17276: Move external getting started to the SDK documentation

  • HANSKEN-17274: Fix broken external links to Hansken website

  • HANSKEN-17278: Add url and license to SDK Python package info

  • HANSKEN-17203: Publish new SDK releases to maven central (Java)

  • HANSKEN-17277: Rearrange code snippets documentation

  • HANSKEN-17273: Update Checkstyle

  • HANSKEN-17214: Fix issues from static code analysis

Release-0.6.0

  • HANSKEN-17196: Downgrade API compatability level to 0.5.0

  • HANSKEN-17194: Update project dependencies to latest versions

  • HANSKEN-16781: Remove the need for plugin=self and moved id to 1st arg when creating a PluginInfo in Python SDK

  • HANSKEN-17191: Add quicklinks to the documentation index

  • HANSKEN-16756: Read extraction plugin version from plugin mavens project.version

  • HANSKEN-16705: Improve apis for declaring plugin resources and querying the data context at runtime

  • HANSKEN-17151: Store Plugin info in plugin image labels for Python plugins

  • HANSKEN-16753: Store Plugin info in plugin image labels for Java plugins

  • HANSKEN-17178: Updated Python tracelet documentation to contain a working example

Release-0.5.1

  • HANSKEN-17141: Fixed $data.type=... matcher when using run_with_hanskenpy (Python)

  • HANSKEN-16908: Added gRPC health service for Python Extraction Plugins

  • HANSKEN-17138: Avoid multiple plugins running on the same hostname:port (Python)

  • HANSKEN-14755: Use Python test framework wrapper with manually started plugins

  • HANSKEN-16905: Added gRPC health service for Java Extraction Plugins

  • HANSKEN-16901: Exclude old guava version to be able to run RemoteExtractionPluginFlits from Intellij

  • HANSKEN-16900: TestTraceSearcher will now return searched traces in natural sorted order of their file names

  • HANSKEN-16725: Improve tracelet api in Python SDK

  • HANSKEN-16764: Corrected ‘Adding data to traces’ code snippets

  • HANSKEN-16704: Use Python’s dataclasses where applicable

  • HANSKEN-17064: Improve runtime packing type checks to allow Sequence / Mapping compatible values

Release-0.5.0

  • HANSKEN-16638: Improve tracelet api in Java SDK

  • HANSKEN-16576: Support vector data type in Python SDK

  • HANSKEN-16707: Update build dependencies, build and test for Python 3.10

  • HANSKEN-16575: Support vector data type in Java SDK

  • HANSKEN-16574: Support vector data type in common SDK

Release-0.4.14

  • HANSKEN-16632: Fix execution of meta-extraction plugins with hansken.py runner which failed with an ‘expected $data in matcher’ error

  • HANSKEN-16634: Allow forward compatibility with Hansken by introducing an GRPC API version

  • HANSKEN-16489: Let build pipeline publish Java artifacts to community

  • HANSKEN-16558: Serve SDK test framework errors in a more developer-friendly way

  • HANSKEN-16489: Removed incompatibility warning for All In One with Hansken.py

  • HANSKEN-16403: Fixed running markdownlint with tox -e markdownlint

  • HANSKEN-16268: Added * value support to the HQL-Lite term matcher, improved documentation by using HQL default property:value instead of property=value

  • HANSKEN-16258: Fixed Jenkins build

  • HANSKEN-16257: Fixed docker stop command in test framework

Release-0.4.13

  • HANSKEN-16229: Fixed SDK documentation not correctly zipped

  • HANSKEN-16095: Documentation: added Test framework data-stream type note

  • HANSKEN-15961: Redundant plugin errors in the client set to log level debug

  • HANSKEN-16111: Updated the testframework to skip search traces when scanning input files

  • HANSKEN-16128: Updated the testframework to disallow overwriting properties, similar to Hansken

  • HANSKEN-16191: Fixed Jenkins build

    • commit id could not be retrieved

    • curl didn’t work due to a bad proxy

  • HANSKEN-16116: Updated FLITS dependency to 3.5.2

  • HANSKEN-16105: Seeking beyond EOF for Python throws an exception

  • HANSKEN-16160: Updated remaining JUnit 4 tests to JUnit 5

  • HANSKEN-16118: Trace ids are now optional when writing automated tests using Flits

  • HANSKEN-16012: Fixed trace types were not correctly determined from property names (Python)

  • HANSKEN-16238: Changed default log level of extraction plugin server. Added command line option to increase it.

Release-0.4.12

  • HANSKEN-15857: Document HQL-lite for programmers manual

  • HANSKEN-16139: Run markdownlint in Jenkins instead of Docker

  • HANSKEN-16115: Updated log4j to version 2.16.0 due CVE-2021-44228

  • HANSKEN-15651: Added documentation on test files

  • HANSKEN-16001: Improved Python exceptions for better readability

  • HANSKEN-16044: Fixed documentation being unstashed to wrong directory on Jenkins

  • HANSKEN-14586: Documentation: added code snippets for adding a Datastream

  • HANSKEN-16038: Hansken.py: fix deferred tools

  • HANSKEN-15653: Added documentation on use with Hansken.py

  • HANSKEN-15801: Change build agent in Jenkins file

  • HANSKEN-16090: Fix for documentation build (m2r2 requires mistune < 2.0.0)

  • HANSKEN-15771: Document trace properties that mismatch the trace model

Release-0.4.11

  • HANSKEN-16048: Fix documentation was not released because the documentation zip was located in docs/_build/docs

Release-0.4.10

  • HANSKEN-16037: Repair release

Release-0.4.9

  • HANSKEN-15858: Add a link to the Extraction Plugin SDK API Javadocs

  • HANSKEN-15656: Documentation: expanded FAQ

  • HANSKEN-15993: Added isVerboseLoggingEnabled() method to ExtractionPluginFlits to enable verbose logging if desired

  • HANSKEN-15801: Change build agent in Jenkins file.

  • HANSKEN-15766: Documentation: Mention the Getting Started guide on gitlab

  • HANSKEN-15765: Added documentation on the ‘Anatomy of a plugin’

  • HANSKEN-15772: Added documentation on tracelets

  • HANSKEN-15770: Added debug documentation (Java and Python)

  • HANSKEN-15773: Added documentation linter (markdownlint)

  • HANSKEN-15964: Added –verbose option to Python test_plugin runner

  • HANSKEN-15913: Use traceUid instead of traceId to read data

Release-0.4.8

  • HANSKEN-15668: Bugfix: prevent search from crashing plugin if results contain traces from different images

Release-0.4.7

  • HANSKEN-15871: Fix for RpcUnixTime & RpcZonedDateTime that were parsed incorrectly in Python

  • HANSKEN-15745: Document naming convention and added convienience method id(domain, category, name) to the PluginInfoBuilder

  • HANSKEN-15790: Change dependency pinning policy

  • HANSKEN-15790: Fix typing issues discovered by upgrading mypy-protobuf

  • HANSKEN-15650: Added documentation on general concepts

  • HANSKEN-15743: Fix Test framework should not limit input files to 2G

  • HANSKEN-15846: Improved hansken.py matcher, instead of $data.type all $data matchers are supported

Release-0.4.6

  • HANSKEN-15711: Changed PluginResources cpu/memory values to floats

  • HANSKEN-15683: Changed the trace enrichment order to ensure transformations are handled before properties

  • HANSKEN-15589: Extend PluginInfo with the ability to specify plugin resources (Python)

  • HANSKEN-15588: Renamed PluginInfo pluginResources() to resources() (Java)

  • HANSKEN-15588: Extend PluginInfo with the ability to specify plugin resources (Java)

Release-0.4.5

deprecated release

Release-0.4.4

deprecated release

Release-0.4.3

  • HANSKEN-15641: Changed PluginId to use forward-slashes instead of backward-slashes

  • HANSKEN-15607: Explicitly fail deferred extraction plugins when building PluginInfo if the number of provided iterations is invalid

Release-0.4.2

  • HANSKEN-15632: Changed PluginInfo.license field to optional, for backwards compatibility

Release-0.4.1

deprecated release

Release-0.4.0

  • HANSKEN-15542: Extended Python PluginInfo with a license name and id consisting of domain, category and name

  • HANSKEN-15466: Extended Java PluginInfo with a license name and id consisting of domain, category and name

  • HANSKEN-15572: Include path as a property to trace type in TraceToJson

  • HANSKEN-15034: Add deferred tools to documentation

  • HANSKEN-15541: Publish SDK documentation as maven artifact

  • HANSKEN-15562: Fix test framework does not deserialize intrinsic properties

  • HANSKEN-15365: Create a FLITS test for deferred extraction plugin

Release-0.3.0

  • HANSKEN-15527: Add api changelogs for version 0.3.0

  • HANSKEN-15393: Rename Transformation to DataTransformation and RangedTransformation to RangedDataTransformation

  • HANSKEN-15391: Add support for ranged transformations (Java only)

  • HANSKEN-15390: Add proto definitions for ranged transformations

  • HANSKEN-15392: Add support for ranged transformations (Python)

Release-0.2.0

  • HANSKEN-15520: Add dedicated page for SDK API changes per language

  • HANSKEN-15515: Move Author and MaturityLevel to PluginInfo module (Python)

  • HANSKEN-15514: Refactor ExtractionContext to DataContext (Java and Python)

  • HANSKEN-15512: Move some internals from the EP python API to runtime module

  • HANSKEN-15511: Include API documentation in SDK dev docs

  • HANSKEN-15505: Cleanup SDK: remove unused position and unrequired datatype

  • HANSKEN-15491: Add compatibility check of remote plugin with current SDK version

  • HANSKEN-15502: Update TestRandomAccessData to accept dataType parameter

  • HANSKEN-15495: Make getData() lazy and replace getAllData() with getDataTypes()

  • HANSKEN-15498: Fixed isort configuration (Python)

  • HANSKEN-15029: Limit gRPC search request count

  • HANSKEN-15274: Add Python API for deferred extraction plugins

  • HANSKEN-15035: Allow python deferred plugins to run through hansken.py.

  • HANSKEN-15288: Add a new Trace subclass for search result Traces

  • HANSKEN-15042: Research and implement requesting data through GRPC for search traces

  • HANSKEN-15139: Allow deferred extraction plugins to process traces

  • HANSKEN-15015: Add option to create deferred extraction plugins using sdk

Release-0.1.8

  • HANSKEN-15236: Read initial chunk of data along with an RPC start request (Python)

  • HANSKEN-15338: Add setting of tracelets

  • HANSKEN-15370: Fixed ArrayOutOfBoundsException when there are bytes left when prefilling the cache

  • HANSKEN-15353: Added testframework exception result validation without or on partial message(startsWith, containsInOrder)

  • HANSKEN-15231: Added caffeine block cache implementation for Java RandomAccessData

  • HANSKEN-15276: Added support for Heterogeneous Maps

Release-0.1.7

  • HANSKEN-15282: Change the way reading data in the python sdk works. More BufferedReader functions are supported, including seeking

Release-0.1.6

  • HANSKEN-15294: Make Java SDK available to the Hansken community

  • HANSKEN-15233: Send initial chunk of data along with the start message (Java only)

  • HANSKEN-15232: Added RandomAccessData cache mechanism with fixed size of 1 MB to speed up large file reads

  • HANSKEN-15193: Improve client-side error message when server-side throws an exception

  • HANSKEN-15237: Added logging before and after processing a trace

  • HANSKEN-15187: Bugfix: Fixed flits traces with nested map properties were being parsed incorrectly

  • HANSKEN-15189: Added default log4j logging when no logging is configured

Release-0.1.5

  • HANSKEN-15192: Added Superpom for Java extraction plugins

  • HANSKEN-15186: Fixed matcher $data.mimeType does not work

Release-0.1.4

  • HANSKEN-14820: Bugfix: “Received a message from server, but processing of the trace has not been started yet”

  • HANSKEN-15059: Set up skeleton documentation for Extraction Plugin devs

  • HANSKEN-15048: Allow ‘workers’ as an optional argument when starting an ExtractionPluginServer

Release-0.1.3

  • HANSKEN-14787: Allow extraction plugins to be routed by HEADER by a proxy/loadbalancer

Release-0.1.2

  • HANSKEN-14923: Make SDK work on Windows

Release-0.1.1

  • HANSKEN-14867: Expanded HQL matcher to support Long & List (intrinsic)properties

Release-0.1.0

  • HANSKEN-14879: Allow SDK releases to be published on PyPI

  • HANSKEN-14738: Shade NFI internal projects into the SDK testframework jar

  • HANSKEN-14841: Bugfix where RpcStringMap wasn’t being unpacked properly in Python, which was discovered during a flits test.

  • HANSKEN-14703: Let Python plugins exit gracefully on SIGTERM

  • HANSKEN-14844: SDK: move serve from test_framework to runtime

  • HANSKEN-14793: Add ExtractionPluginBuilder.add_data method in python API

  • HANSKEN-14777: Add an extra check to ignore and log unsupported types during RpcStart gRPC serialization.

  • HANSKEN-14739: License: Distribute Extraction Plugin SDK under the Apache License 2.0

  • HANSKEN-14763: Bugfix where some python plugins were not loaded correctly when using serve or test-plugin commands

  • HANSKEN-14737: Move serve.py from plugin examples to SDK

  • HANSKEN-14720: Add option to use with when using the trace.open method in python

  • HANSKEN-14582: Add option to write data using the python api

  • HANSKEN-14660: Move _test.py files from plugin examples repo to SDK repo

  • HANSKEN-14618: Add validation for unexpected extra data streams to test framework

  • HANSKEN-14704: Fix shading of the runtime super pom

  • HANSKEN-14619: Allow propagation of IOException in plugin new child callback

  • HANSKEN-14632: Add Java gRPC support for writing raw data streams on a trace

  • HANSKEN-14591: Split into three modules in the SDK

  • HANSKEN-14580: Add proto message definitions for raw data stream writing

  • HANSKEN-14635: Trace format which containes name/id can now be deserialized by testframework

  • HANSKEN-14131: Added ‘verbose’ logging for test-framework HQL matching

  • HANSKEN-14130: Updated StandaloneTestRunner to expose more errors & exceptions

  • HANSKEN-13784: Add meta support to test-framework

  • HANSKEN-14581: Extend Trace API with raw data writing capabilities

  • HANSKEN-14547: Deploy Java sources JAR for improved client debugging

  • HANSKEN-14531: Fix Python release. Python needs only one build step, which is either a snapshot or a release build. The separate Python release step was removed and merged with the first Python build step. Repository paths were corrected for the release version.

  • HANSKEN-14531: Fix python release - The python release is no longer a separate step in the build pipeline, since there is no actual difference between a snapshot and a release, apart from the version numbering scheme and the test-framework is downloaded from a repository location depending on the release parameter (see comments in Jenkinsfile)

  • HANSKEN-14161: Add Python test-framework wrapper around Java test-framework and add test-framework.tgz to whl

  • HANSKEN-14234: Restructure build pipeline to build and release Java first

  • HANSKEN-13799: Extraction Plugin: support meta extraction

  • HANSKEN-14286: Create adapter from RandomAccessData to InputStream

  • HANSKEN-14314: Don’t send child name when sending enrichment message

  • HANSKEN-14318: Flush cached children before flushing root in case of error with python gRPC server

  • HANSKEN-13414: Generate shaded jar for runtime that shades Guava, Protobuf, gRPC, and Netty (fix)

  • HANSKEN-14283: Allow passing a configuration of retry policy for the extraction plugin client

  • HANSKEN-14234: Restructure build pipeline to build and release Java first

  • HANSKEN-13414: Generate shaded jar for runtime that shades Guava, Protobuf, gRPC, and Netty

  • HANSKEN-14234: Restructure build pipeline to build and release Java first

  • HANSKEN-14135: Improve testing Python plugins in integration step and test reading large chunks

  • HANSKEN-14134: Fix releasing python plugins

  • HANSKEN-14128: Validate gRPC message limit for Python server instances

  • HANSKEN-14122: Fix missing comma in dependencies which broke the release

  • HANSKEN-14092: The type and total size of the data currently being processed can now be retrieved from an extractioncontext object passed to the process function

  • HANSKEN-13668: Added support for lists of longs, Hansken maps and LatLong to Java and Python API

  • HANSKEN-14104: Set Python gRPC limit to 64 MB(including message overhead)

  • HANSKEN-14079: Add logging to the SDK

  • HANSKEN-14010: Add support for serializing datetime in python API

  • HANSKEN-14083: Fix releasing python sdk

  • HANSKEN-14035: Add static type checks to python project

  • HANSKEN-14030: Allow test framework to be executed standalone for non-java extraction plugins

  • HANSKEN-14073: Support gRPC extraction plugins in test framework

  • HANSKEN-14074: Add support for serialization of Maps.

  • Hansken-14090: Use new Hansken python-api children call for creating nested children

  • HANSKEN-13774: Add support for creating children in python API

  • HANSKEN-13776: Add error handling to the Python based server and send error messages to the client

  • HANSKEN-14060: Remove mapping-interface from python extraction for API consistency

  • HANSKEN-13773: Update trace properties in Python API

  • HANSKEN-14044: Make sure python testing code is linted as well, enforce single quotes

  • HANSKEN-13772: Expose trace properties in Python API

  • HANSKEN-13775: Python - added trace.open() functionality to read from data streams

  • HANSKEN-14031: Split Trace interfaces (hansken.py trace vs external plugin trace)

  • HANSKEN-14037: Make sure pytest is always used for python tests

  • HANSKEN-14011: Implement unpack for trace

  • HANSKEN-14009: Implement pack for trace and trace enrichment

  • HANSKEN-14008: Move generated hql-lite parsers to different package (conflicts with hql package)

  • HANSKEN-13777: Added utility to run Python Extraction Plugin implementations with Hansken.py

  • HANSKEN-13771: Implement Extraction Plugin Info for Python plugins

  • HANSKEN-13966: Give socketproxy disconnect some time to disconnect (fixes flaky unit test)

  • HANSKEN-13810: Add extraction plugin python server code

  • HANSKEN-13676: Added support for ZonedDateTime over gRPC

  • HANSKEN-13922: Add webpage url to PluginInfo

  • HANSKEN-13676: Changed the way of creating child Traces to using a consumer.

  • HANSKEN-13655: Added server/client disconnect tests and implemented initial handling server-side

  • HANSKEN-13809: Added missing gRPC exception handles of process()

  • HANSKEN-13801: Made HQL-Lite matchers immutable

  • HANSKEN-13761: Seperated HQL-Lite type matcher implementation

  • HANSKEN-13756: Added HQL-Lite datastream matchers

  • HANSKEN-13800: (Temporarily) remove meta from Extraction Plugin API

  • HANSKEN-13798: Propagate exception on failure of START serialization

  • HANSKEN-13706: Create basic test framework implementation

  • HANSKEN-13769: Make jenkins run tests and a linter for python

  • HANSKEN-13705: Add support for creating child traces over gRPC

  • HANSKEN-13713: Make (partial) extraction plugin errors visible for clients

  • HANSKEN-13709: Send partial result when an external plugin errors out

  • HANSKEN-13733: Add script to generate gRPC Python files

  • HANSKEN-13660: Copied the Hql definition from Hansken to enable the HQL-Lite implementation

  • HANSKEN-13656: Test and handle invalid protocol messages

  • HANSKEN-13663: Add matcher interface to PluginInfo

  • HANSKEN-13714: Add IOException to ExtractionPlugin.process() interface

  • HANSKEN-13658: test(s) for non-grpc connection with a grpc server

  • HANSKEN-13650: Add basic support for writing trace information over gRPC

  • HANSKEN-13648: Add basic support for reading trace information over gRPC

  • HANSKEN-13651: Default implementations for RandomAccessData interface

  • HANSKEN-13643: Add basic support for reading trace information over gRPC

  • HANSKEN-13643: Add basic support for reading from trace data over gRPC

  • HANSKEN-13581: Plugin-info client/server implementations

  • HANSKEN-13580: Add gRPC server and client to serve an Extraction Plugin as a service

  • HANSKEN-13579: Created initial gRPC implementation messages

  • HANSKEN-13577: Be able to create releases of the SDK

  • HANSKEN-13578: Created plugin API

  • HANSKEN-13560: Create initial gRPC definitions

  • HANSKEN-13554: Initial repository