Setup

• 1: Prerequisites
• 2: fat-jar
• 3: fat-jar (TLS)
• 4: Ports
• 5: Metrics
• 6: Compression
• 7: Security
• 8: gRPC Client
• 8.1: gRPC BasicAuth
• 8.2: gRPC KeepAlive
• 8.3: gRPC Resilience
• 9: Properties
• 10: Docker

1 - Prerequisites

Postgres

In order to run the FactCast server, you have to provide a Postgres database at least in version 9.4. The following example shows the configuration with one user.

spring.datasource.username="user" //that user has to be provided
spring.datasource.password="password"

The user has to be a superuser, as he will also install Postgres modules like uuid-ossp.

If you don’t want to provide a superuser

If you don’t want to provide a superuser, you have to consider the following points:

1.) The database user needs at least the permission to query the pg_roles view. According to the documentation it’s publicly accessible, so that shouldn’t be a problem.

2.) The FactCast needs the Postgres module uuid-ossp. You have to install that module manually. The server will recognize the already installed module and it won’t throw an error caused by missing privileges. Login into your Postgres console and execute the following command as superuser:

CREATE EXTENSION IF NOT EXISTS "uuid-ossp";

2 - fat-jar

Building

First of all, build factcast completely if not yet done by running

mvn install

In order to run a simple example FactCast server, you could enter the project factcast-examples/factcast-example-server and run

mvn spring-boot:run

or run it in your IDE. Note that it will use TestContainer to start an ephemeral postgres instance for you. That means, you need to have a runnable Docker installed on your machine.

In case you want to use your local Postgres instead, take a look at ExampleServerWithPostgresContainer to find out how what is necessary to use a pgsql. After all, this is just a very simple Spring Boot application using JDBC.

As expected, running

mvn package

will create a standard spring boot fat jar target/factcast.jar that can be run instantly.

Read more on Ports

3 - fat-jar (TLS)

Non-Encrypted

In order to run a simple example FactCast server, you could enter the project factcast-examples/factcast-example-server, see fatjar

TLS Server

There is an extra example project that demonstrates the usage of TLS for your server that can be found factcast-example-tls-server

We tried to stick as close as possible to what we have in factcast-examples/factcast-example-server to demonstrate the necessary changes and nothing more.

Obviously, for running a TLS Server, you need a certificate. We packaged a snakeoil localhost certificate for you to test. This cert can be found in src/etc/certificates/. In order to create your own selfsigned certificate, there is a shell script you can use as a starting point.

obviously, you should use proper trusted certificates when you run FactCast in production - you have been warned

In order to run the TLS Server, go to factcast-examples/factcast-example-tls-server and run

mvn spring-boot:run

Read more on Ports

4 - Ports

The default TCP-Port exposed is 9090. As usual, you can set it via environment variables.

Standard ports used:

5 - Metrics

Being a regular Spring Boot 2+ application, the FactCast Server uses micrometer.io as its metrics emitting/collecting solution. In order to get started collecting the metrics FactCast Server emits, you’ll need to choose a backend/store for your metrics. Micrometer has lots of prebuilt bindings to choose from. Please refer to the respective documentation in the Setup section of the micrometer docs.

When it comes to metrics, you’ll have to know what you’re looking for. There are

• Server metrics in FactCast Server as well as
• Client metrics in the factcast client and additionally in the
• factus client library.

We’re focussing on Server metrics here.

Metric namespaces and their organization

At the time of writing, there are six namespaces exposed:

• factcast.server.timer
• factcast.server.meter
• factcast.store.timer
• factcast.ui.timer
• factcast.store.meter
• factcast.registry.timer
• factcast.registry.meter

Depending on your micrometer binding, you may see a slightly different spelling in your data (like ' factcast_store_timer`, if your datasource has a special meaning for the ‘.’-character)

Furthermore, metrics in operations are automatically tagged with

• an operation name
• a store name (‘pgsql’ currently) and
• an exception tag (‘None’ if unset).

Existing metrics

There are a bunch of metrics already emitted in the server. These metrics can be grouped by type:

• Timers (collecting durations of code execution)
• Meters (collecting metric events, for example, occurrences of errors)

As this list is continuously growing, we cannot guarantee the documentation’s completeness. If you want to see the current list of operations, please look at StoreMetrics.java , RegistryMetrics.java , ServerMetrics.java, or UIMetrics.java respectively.

At the time of writing (0.4.3), the metrics exposed by the namespaces group factcast.server are:

At the time of writing (0.4.3), the metrics exposed by the namespaces group factcast.store are:

At the time of writing (0.4.3), the metrics exposed by the namespaces group factcast.registry are:

gRPC Metrics

If you’re looking for remote calls and their execution times (including marshalling/de-marshalling from protobuf), you can have a look at the metrics automatically added by the gRPC library that we use. The relevant namespaces are:

• grpcServerRequestsReceived and
• grpcServerResponsesSent

These automatically added metrics only focus on service methods defined in the protocol buffer specs. Since a gRPC remote call triggers not everything we want to measure, we introduced additional metrics. When comparing, for instance, the automatically added durations of gRPC vs. the ‘factcast.store.duration’, you will find a subtle difference. The reason for this is that instead of including the gRPC overhead, we chose to only measure the actual invocations on the FactStore/TokenStore implementation. Depending on your needs, you may want to focus on one or the other.

Executor Metrics

Micrometer provides an integration to monitor the default thread pool executor created by Spring Boot. Under the same namespace executor.*, we publish metrics for our own thread pool executors used inside FactCast.

You can distinguish them by the name tag. Currently, these are:

• subscription-factory - used for incoming new subscriptions
• fetching-catchup - used for buffered transformation while using the fetching catchup strategy
• paged-catchup - used for buffered transformation while using the paged catchup strategy
• transformation-cache - used for inserting/updating entries in the transformation cache (only if you use persisted cache)

See https://micrometer.io/docs/ref/jvm for more information.

UI Metrics

Special metrics for the FactCast-Server UI are published via factcast.ui.timer namespace.

Additionally, all methods of the org.factcast.server.ui.adapter.FactRepositoryImpl are measured time-wise, and can be visualized via class and method dimension.

6 - Compression

Why ?

Wherever there is network communication, the question of compression comes up. The FactCast server currently supports three compressors out of the box:

• LZ4
• Snappy
• GZip

Unfortunately, GRPC does not support stream-compression, but only message-compression., This means that the efficiency of the compression is dependent on the message size. We’ll get to that…

Client chooses

In order to agree on which compressor to use, there is an initial handshake when the client connects to the server, in which the available compressors on client and server are compared, and the server selects the one to use.

The server send a list of what he accepts, and the client picks his favorite compressor out of that list in the order shown above (LZ4 first, then snappy, GZip as a fallback).

As the client should be low on dependencies and assumptions, Gzip (as supported by the JDK) is the default compressor every client supports.

In order to prefer snappy or LZ4, you’d need to add one or both of the following dependencies (or later versions) to your client. Once they are on the classpath, the client will pick them up automatically, and the server will prefer them over GZip.

<dependency>
  <groupId>org.xerial.snappy</groupId>
  <artifactId>snappy-java</artifactId>
  <version>1.1.8.4</version>
</dependency>

<dependency>
  <groupId>net.jpountz.lz4</groupId>
  <artifactId>lz4</artifactId>
  <version>1.3.0</version>
</dependency>

Compressor efficiency

As there currently is no stream-compression in GRPC, the server compresses each message transferred to the client separately. The smaller this message is, the less efficient the compression can be. For this reason it is important (during the catchup phase, where you expect a lot of messages) to allow the server to bundle messages into a batch that it will compress and send as one message.

See factcast.grpc.client.catchup-batchsize

In the follow phase, this setting has no meaning, as you don’t want to wait for your batch to fill up before you receive the latest publications from the server. Latency is more important than compression efficiency in that case.

7 - Security

Authentication & Authorization

In order to control access the FactCast Server supports a very basic way of defining which client is allowed to do what. The security feature is enabled by default, but can be disabled (for integration tests for example) by setting factcast.security.enabled to false.

In order to make use of the security features, a Bean of type FactCastAccessConfig must be defined. This is done either by providing one in your FactCast Server’s context, or by using the dead-simple approach to put a factcast-access.json on the root of your classpath or at /config/ to deserialize it from there.

Example below.

Now, that you’ve defined the access configuration, you also need to define the secrets for each account. Again, you can do that programmatically by providing a FactCastSecretsProperties, or by defining a property for each account like this:

factcast.access.secrets.brain=world
factcast.access.secrets.pinky=narf
factcast.access.secrets.snowball=grim

The catch with this simple approach of course is, that credentials are stored in plaintext in the server’s classpath, but remember it is just a dead-simple approach to get you started. Nobody says, that you cannot provide this information with a layer of your docker container, pull it from the AWS Parameter Store etc…

If FactCast misses a secret for a configured account on startup, it will stop immediately. On the other hand, if there is a secret defined for a non-existing account, this is just logged (WARNING-Level).

The contents of factcast-access.json might look like:

{
	"accounts": [
		{
			"id": "brain",
			"roles": ["anything"]
		},
		{
			"id": "pinky",
			"roles": ["anything", "limited"]
		},
		{
			"id": "snowball",
			"roles": ["readOnlyWithoutAudit"]
		}
	],
	"roles": [
		{
			"id": "anything",
			"read": {
				"include": ["*"]
			},
			"write": {
				"include": ["*"]
			}
		},
		{
			"id": "limited",
			"read": {
				"include": ["*"],
				"exclude": ["secret"]
			},
			"write": {
				"exclude": ["audit*"]
			}
		},
		{
			"id": "readOnlyWithoutAudit",
			"read": {
				"include": ["*"],
				"exclude": ["audit*", "secret"]
			},
			"write": {
				"exclude": ["*"]
			}
		}
	]
}

Where pinky & brain are authorized to use the full FactStore’s functionality (with ‘pinky’ not being able to write to namespaces that start with ‘audit’) whereas snowball can only read everything but ‘audit’-namespaces, but not write anything.

In case of conflicting information:

• explicit wins over implicit
• exclude wins over include

Note, there is no fancy wildcard handling other than a trailing ‘*’.

see module examples/factcast-example-server-basicauth for an example

Using BasicAuth from a client

From a client’s perspective, all you need to do is to provide credentials. Once the credentials are configured, they are used on every request in a Basic-Auth fashion (added header to request).

In order to define credentials, just set the appropriate property to a value of the format ‘username:password’, just as you would type them into your browser when a basic-auth popup appears.

# if this property is set with a value of the format 'username:password', basicauth will be used.
grpc.client.factstore.credentials=myUserName:mySecretPassword

You can always use environment variables or a -D switch in order to inject the credentials.

see module examples/factcast-example-client-basicauth for an example

Customizing Credential Loading

If you dont want to configure your passwords via properties, you can provide either a custom FactCastSecretProperties bean or an implementation of a UserDetailsService. That’s a simple interface coming from Spring Security which provides a mapping method from username to user. In our case we have to return a FactCastUser.

If you want to externalize secret loading but want to keep the factcast-access.json file for managing authorization an implementation of such a UserDetailsService could look like this:

@Bean
UserDetailsService userDetailsService(FactCastAccessConfiguration cc, PasswordEncoder passwordEncoder) {
    return username -> {
        // fetching account info from fact-access.json
        Optional<FactCastAccount> account = cc.findAccountById(username);

        // your way to fetch the user + password
        User user = loadUserByName(username);

        return account
            .map(acc -> new FactCastUser(acc, passwordEncoder.encode(user.getPassword())))
            .orElseThrow(() -> new UsernameNotFoundException(username));
    };
}

8 - gRPC Client

GRPC Clients

In order to talk to a - not in process - factstore (which is the usual setup for non-test applications), GRPC is the communication protocol used.

Using FactCast client in Spring boot via GRPC

If you use Spring take the easy path in your Spring Boot Application by adding the appropriate dependencies to your application:

   <dependency>
     <groupId>org.factcast</groupId>
     <artifactId>factcast-client-grpc</artifactId>
   </dependency>
   <dependency>
     <groupId>org.factcast</groupId>
     <artifactId>factcast-spring-boot-autoconfigure</artifactId>
   </dependency>

There are example projects: factcast-examples/factcast-example-client-spring-boot2 and factcast-examples/factcast-example-client-spring-boot1 respectively, that you can use as a template.

Note that factcast-client-grpc is built on top of (https://github.com/yidongnan/grpc-spring-boot-starter). If you are looking for the basic configuration properties, that is where you can find the latest version.

At the time of writing, the most relevant are:

8.1 - gRPC BasicAuth

Using BasicAuth from a client

From a client’s perspective, all you need to do is to provide credentials. Once the credentials are configured, they are used on every request in a Basic-Auth fashion (added header to request).

In order to define credentials, just set the appropriate property to a value of the format ‘username:password’, just as you would type them into your browser when a basic-auth popup appears.

# if this property is set with a value of the format 'username:password', basicauth will be used.
grpc.client.factstore.credentials=myUserName:mySecretPassword

You can always use environment variables or a -D switch in order to inject the credentials.

see module examples/factcast-example-client-basicauth for an example

8.2 - gRPC KeepAlive

keep-alive settings

Here are some good settings for an initial configuration of a SpringBoot FactCast client/server setup in case you ran into gRPC related client server communication troubles.

• Sending keep-alive HTTP/2 PINGs on the connection is useful in case you are running on infrastructure that doesn’t support configurable idle timeouts, and therefore closes connections.

• The proposed values are defining a scenario where the client sends keep-alive HTTP/2 PINGs every 300s and the server accepts this behavior without sending GO_AWAY ENHANCE_YOUR_CALM to the client. Please adapt to your specific needs.

Client side

Further details can be found here : net.devh.boot.grpc.client.config.GrpcChannelProperties.

Server side

Further details can be found here : net.devh.boot.grpc.server.config.GrpcServerProperties.

8.3 - gRPC Resilience

Resilience approach

In order to make it easier for clients to deal with errors, we try to mitigate connection or network errors, or RetryableExceptions in general by just retrying. There are two types of gRPC communications within the FactCast gRPC API:

• synchronous, request / response
• asynchronous, request / streaming response

While the first can be mitigated easily by retrying the call, things get more complicated in an asynchronous, streaming scenario. Imagine a subscription to particular facts (let’s say 10) from scratch, where after 5 successfully received facts the network connection fails. Now simply retrying would mean to receive those 5 facts again, which is not only wasteful, but also hard to handle, as you’d need to skip those rather than process them a second time. Here the FactCast gRPC client keeps track of the facts successfully processed and resubscribes to the ones missing. In this example, it’ll try to subscribe to the same factstream but starting after the fifth fact.

Resilience is supposed to “just work” and let you deal with just the non-transient errors. This is why it is enabled by default with sane defaults.

If you want to disable it completely for any reason, you always can use

factcast.grpc.client.resilience.enabled=false

See properties for the defaults.

9 - Properties

Schemaregistry

Transformation-Registry

Performance / Reliability

Snapshots

RedisSnapshots

gRPC

Properties you can use to configure gRPC:

gRPC Client

gRPC Client recommended settings

grpc.client.factstore.enable-keep-alive=true
grpc.client.factstore.keep-alive-time=300
grpc.client.factstore.keep-alive-without-calls=true

Further details can be found here : net.devh.boot.grpc.client.config.GrpcChannelProperties.

FactCast client specific

grpc Server

gRPC Server recommended settings

grpc.server.permit-keep-alive-without-calls=true
grpc.server.permit-keep-alive-time=100

Blacklist

Testing

UI

Further details can be found here : net.devh.boot.grpc.server.config.GrpcServerProperties.

10 - Docker

Building

In order to build a standard docker container from source, enter the project factcast-docker and run

mvn docker:build

This will create a docker container as factcast/factcast and needs nothing more than the database URL to run.

Usage:

The docker container can be started

docker run -e"SPRING_DATASOURCE_URL=jdbc:postgresql://<POSTGRES-SERVER>/<DATABASENAME>?user=<USERNAME>&password=<PASSWORD>" -p 9090:9090 factcast/factcast

Note, that the resulting server is optimized and supposed to be used for integration testing only. Do not use it in production