Code organization

• 1 Core libraries
  • 1.1 libutils
  • 1.2 libdata
  • 1.3 libflow
  • 1.4 libquery
  • 1.5 libgather
  • 1.6 libaggregator
• 2 Extension interfaces
• 3 Extension implementations
• 4 The main application
• 5 The test helpers

There's API documentation for the types and functions in code. However, the aggregator's code base is quite large and knowing what to look for where is useful. This document describes somewhat higher overview of the code organization into areas of responsibility.

The code is split into multiple Rust crates (eg. packages), each residing in its own directory. These work both as organization units and units of compilation. Some of them might be optional (eg. data sources and internal computations might be turned on or off through Cargo features).

Obviously, there are many ways how to organize code into areas and the current state might not be the best one possible (both subjectively and objectively), but something must have been chosen.

1 Core libraries

These form the foundation and generic parts of aggregator (eg. they are always compiled in).

1.1 libutils

This library holds various utilities of different purposes, but with wide-spread use. Therefore, things like logging and command line parsing lives here.

1.2 libdata

All kinds of small bits and pieces that are widely used live here ‒ data types describing times or statistics, for example. While they have some methods and support implementations (eg. they might know how to be serialized), they are more of building blocks than holding some kind of logic.

1.3 libflow

The library takes the data types from libdata and builds the abstraction of the whole flow from them. It also adds some other related types (eg. the Update that describes addition of information into a flow).

1.4 libquery

This describes the user-facing query and the response. It knows how to deserialize the query and serialize the response. However, it does not know how to perform the query, only how a query looks like.

1.5 libgather

The Gather type that lives here takes care of the live flows and applying updates onto them. This is where data from multiple data sources meet, where the time is split into slices, etc.

1.6 libaggregator

This is the top-level library of the aggregator. It contains the Reactor, which is an event loop that coordinates asynchronous execution of tasks and IOs. It also contains the Keeper, which is responsible for keeping already closed flow slices, running queries on them and (in the future) managing their on-disk storage.

2 Extension interfaces

The dsrc and int-compute libraries hold the interface for the corresponding extensions (data sources and internal computations). These interfaces are used by the core libraries to manipulate the extensions in a generic ways, without any knowledge about the concrete implementations.

3 Extension implementations

The libraries prefixed with dsrc or int-compute hold the actual implementations of the extensions. These are depended on by the main application and plugged into the core Reactor on startup, based on compile-time and run-time configuration.

4 The main application

The top-level crate is the application. It is quite a thin wrapper around libaggregator, which only plugs the movable parts together and starts it up.

5 The test helpers

There are some test-help libraries that are used for comfortable unit test writing. These are not part of the final product.