Joule Release 1.0.4

Introduction

Joule is a Low Code Platform for use case development. The platform brings simplicity to developing use cases by providing an expressive language to define processing pipelines using prebuilt and custom processors and data connectors. Out-of-the-box Joule provides standard data connector implementations, and useful processors that enables you to start building and running use cases quickly.

Key Features

Feature Engineering

Joule includes a feature engineering processor that allows users to define how features should be created for predictive analytics use cases.

For each declared feature field, the processor generates an engineered value using one of two supported methods:

• Raw values
• Computed values using custom expressions or plugins

Once processing is complete, a feature map containing all required features is generated and embedded into the StreamEvent, ready for the next processor in the pipeline.

To help you get started, Joule provides out-of-the-box (OOTB) plugins across the following functional categories:

• Scripting
• Scaling
• Transform

Example

feature engineering:
  name: retailProfilingFeatures
  versioned: true
  features:
    as values:
      - location_code
      - store_id

    compute:
      spend_ratio:
        scripting:
          macro:
            expression: 1 - spend/avg_spend
            language: js
            variables:
              avg_spend: 133.78
      age:
        function:
          age binning:
            source field: date_of_birth
      day:
        function:
          day-of-week transform:
            source field: date

See documentation for further details.

Machine Learning & Auditing

Joule provides a PMML predictor processor to perform streaming predictions / scoring. The implementation leverages the JPMML open source library developed by Villu Ruusmann.

Example

pmml predictor:
  name: irisScorer
  model filename: /hom/joule/models/pmml/iris_rf.pmml
  response field: flowerPrediction
  audit configuration:
    target schema: ml_audit
    queue capacity: 5000
    flush frequency: 5 

Auditing

An optional configuration enables auditing of predictions to support model retraining, monitor feature and prediction drift, ensure model observability, and meet local governance requirements.

When enabled, this configuration dynamically creates an in-memory database table using the process name as the table name, and exposes REST endpoints for direct access and export.

With these capabilities, Joule now supports stream-based advanced analytics using PMML models within Docker containers. An example is shown below.

See documentation for further details.

SQL Query API

Joule embeds DuckDB, an in-memory database, in to the runtime process. The solution is ideal for supporting custom processor logic using various methods such as:

• Hosting and accessing custom reference data
• Scratchpad for stateful processing
• Ad-hoc custom complex queries
• Capture and exporting streaming events

See documentation for further details.

Websocket Support

Joule now supports publishing events on to a Websocket publisher. Events are serialised as Json.

websocketPublisher:
  pathOverride: /joule/websocket/stream

See documentation for further details.

Project Templates

To help you get started with custom transport and processor development, a project template is available here.

Documentation

As part of our ongoing commitment to high-quality documentation, this release includes a major reorganisation, new sections, expanded details, and many other improvements. You can find the updated documentation online here.

Getting started

To get started, download the following resources to prepare your environment and work through the provided documentation. Feel free to reach out with questions.

• Download the examples code from GitLab
• Pull the latest docker image from DockerHub
• Work through the README file

We’re here to help

Feedback is most welcome including thoughts on how to improve and extend Joule and ideas for exciting use cases.

Join the FractalWorks Community Forum, who openly share ideas, best practices and support each other. Feel free to join us there! And if you have any further questions on how to become a partner or customer of FractalWorks, do not hesitate to engage with us, we will be happy to talk about your needs.