Ops

• 1: Metrics
• 2: Telemetry
• 3: General Operations Tips

1 - Metrics

Being a regular Spring Boot 2+ application, the FactCast Server uses micrometer.io as its metrics emitting/collecting solution. In order to get started collecting the metrics FactCast Server emits, you’ll need to choose a backend/store for your metrics. Micrometer has lots of prebuilt bindings to choose from. Please refer to the respective documentation in the Setup section of the micrometer docs.

When it comes to metrics, you’ll have to know what you’re looking for. There are

• Server metrics in FactCast Server as well as
• Client metrics in the factcast client and additionally in the
• factus client library.

We’re focussing on Server metrics here.

Metric namespaces and their organization

At the time of writing, there are six namespaces exposed:

• factcast.server.timer
• factcast.server.meter
• factcast.store.timer
• factcast.ui.timer
• factcast.store.meter
• factcast.registry.timer
• factcast.registry.meter

Depending on your micrometer binding, you may see a slightly different spelling in your data (like ' factcast_store_timer`, if your datasource has a special meaning for the ‘.’-character)

Furthermore, metrics in operations are automatically tagged with

• an operation name
• a store name (‘pgsql’ currently) and
• an exception tag (‘None’ if unset).

Existing metrics

There are a bunch of metrics already emitted in the server. These metrics can be grouped by type:

• Timers (collecting durations of code execution)
• Meters (collecting metric events, for example, occurrences of errors)

As this list is continuously growing, we cannot guarantee the documentation’s completeness. If you want to see the current list of operations, please look at StoreMetrics.java , RegistryMetrics.java , ServerMetrics.java, or UIMetrics.java respectively.

At the time of writing (0.4.3), the metrics exposed by the namespaces group factcast.server are:

At the time of writing (0.7.6), the metrics exposed by the namespaces group factcast.store are:

At the time of writing (0.4.3), the metrics exposed by the namespaces group factcast.registry are:

gRPC Metrics

If you’re looking for remote calls and their execution times (including marshalling/de-marshalling from protobuf), you can have a look at the metrics automatically added by the gRPC library that we use. The relevant namespaces are:

• grpcServerRequestsReceived and
• grpcServerResponsesSent

These automatically added metrics only focus on service methods defined in the protocol buffer specs. Since a gRPC remote call triggers not everything we want to measure, we introduced additional metrics. When comparing, for instance, the automatically added durations of gRPC vs. the ‘factcast.store.duration’, you will find a subtle difference. The reason for this is that instead of including the gRPC overhead, we chose to only measure the actual invocations on the FactStore/TokenStore implementation. Depending on your needs, you may want to focus on one or the other.

Executor Metrics

Micrometer provides an integration to monitor the default thread pool executor created by Spring Boot. Under the same namespace executor.*, we publish metrics for our own thread pool executors used inside FactCast.

You can distinguish them by the name tag. Currently, these are:

• subscription-factory - used for incoming new subscriptions
• parallel-transformation - used for batch transformation of buffered facts
• paged-catchup - used for buffered transformation while using the paged catchup strategy
• transformation-cache - used for inserting/updating entries in the transformation cache (only if you use persisted cache)
• pg-listener - used by the Guava EventBus that receives signals from the PostgreSQL
• telemetry - used by the Guava EventBus that receives signals from the FactCast Server ( see telemetry)

See https://micrometer.io/docs/ref/jvm for more information.

UI Metrics

Special metrics for the FactCast-Server UI are published via factcast.ui.timer namespace.

Additionally, all methods of the org.factcast.server.ui.adapter.FactRepositoryImpl are measured time-wise, and can be visualized via class and method dimension.

2 - Telemetry

Starting from factcast version 0.7.9, you can extend your server implementation to listen for internal telemetry events. This can be useful for monitoring and debugging purposes.

The telemetry events are emitted using a dedicated internal Guava EventBus.

Subscription lifecycle events

Currently, the factcast-store module emits an event on each phase of the subscription lifecycle (see org.factcast.store.internal.telemetry.PgStoreTelemetry):

• PgStoreTelemetry.Connect emitted whenever a client connects to the factcast server
• PgStoreTelemetry.Catchup emitted whenever the subscription catches up to the current state of the store
• PgStoreTelemetry.Follow emitted whenever the subscription started consuming live events
• PgStoreTelemetry.Close emitted whenever the client disconnects from the factcast server
• PgStoreTelemetry.Complete emitted whenever the subscription completed its lifecycle

Each emitted event contains a request, which holds the client’s request details.

How to listen to telemetry events

It boils down to implementing a listener that is able to consume telemetry events, through com.google.common.eventbus.Subscribe annotated methods, and registering it via the PgStoreTelemetry bean.

Here is an example:

import com.google.common.eventbus.Subscribe;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.factcast.store.internal.telemetry.PgStoreTelemetry;

@RequiredArgsConstructor
@Slf4j
public class MyTelemetryListener {

  public MyTelemetryListener(PgStoreTelemetry telemetry) {
    telemetry.register(this);
  }

  @Subscribe
  public void on(PgStoreTelemetry.Connect signal) {
    log.info("FactStreamTelemetry Connect: {}", signal.request());
  }

  @Subscribe
  public void on(PgStoreTelemetry.Close signal) {
    log.info("FactStreamTelemetry Close: {}", signal.request());
  }
}

You can check out the full example in the factcast-example-server-telemetry module. That module contains a simple example of how to listen to each subscription lifecycle event, to log the request details and maintaining a list of following subscriptions, which can be read through the actuator /info endpoint.

3 - General Operations Tips

Conditional execution date2serial migration

In FactCast v0.7.1, a new UI feature was introduced to allow filtering of events based on their publishing date. For this purpose, the date2serial mapping table was introduced in the schema.

A Liquibase changeset takes care of creating and populating the date2serial table, but it is not executed when the store contains more than 10 million events. This is to prevent the migration from taking too long on larger setups.

As mentioned in the changeset comments, it is suggested to run the changeset manually in such cases. The changeset can be found in the factcast-store module under src/main/resources/db/changelog/factcast/issue2479/date2serial_for_existing_events.sql.

Optimize GIN indexes updates

While GIN Indexes make querying jsonb faster they are also expensive to update. Especially because a single change can cause the update of multiple index entries. In order to keep the overhead on write and update statements low, postgres per default enables the fastupdate setting which defers the update of the index and instead gathers changes to execute them all at once. This update happens when:

• the gin_pending_list_limit is reached (default 4MB)
• the gin_clean_pending_list function is called
• at the end of the autovacuum of the table

This can cause the query whose change eventually fills the gin_pending_list_limit to be a lot slower than usual. If this kind of behavior is observed it might make sense to consider to:

• reduce the gin_pending_list_limit -> more frequent, smaller flushes
• increase the limit and do manual flushes outside of workload
• turn off fastupdate
• let autovacuum run more often or manually call the clean operation

Autoanalyse & Autovacuum settings

Postgres has a built-in mechanism to keep the statistics up to date and to clean up dead tuples, called Automatic Vacuuming.

In most cases, it might be necessary to adjust the default autovacuum settings to better fit the workload and ensure a more efficient execution of the process:

# disable autovacuum schedule based on scale factor
autovacuum_vacuum_scale_factor:0
autovacuum_analyze_scale_factor:0
# set thresholds based on approximate number of facts inserted
autovacuum_vacuum_threshold:<number of new facts each month>
autovacuum_analyze_threshold:<number of new facts each week>

AWS RDS Configuration

Most of the time, the default RDS configuration of PostgreSQL is sufficient. However, in some cases, it might be necessary to adjust some settings in the RDS Parameter Groups to improve performance. The following settings are recommended for FactCast instances running on production stages:

# hands over concurrency considerations to kernel
effective_io_concurrency:0
# tune accordingly, consider roughly 100mb running on a db.r5.2xlarge RDS instance
work_mem:100000
# the followings might vary, depending on your non-functional requirements
log_statement:'none'
log_min_duration_statement:500
default_statistics_target:100