Rai Multicast Services Guide

Rai MS is a Link State based protocol for the construction of Pub/Sub messaging systems which allows for loops and redundancies in the network connections between peers. It has 4 different types of network transports:

The first 3 transports may be interconnected with redundancies. The local bridging transports strips or adds the meta-data of the message that allows for routing through the network, so it can’t be looped.

It uses a best effort delivery system. It serializes messages based on subject so that streams are delivered in order discarding duplicates, but messages which are lost in transit because of node or network failures, are not retransmitted.

Distributed systems are more often crossing network boundaries. Traditional broker based systems or multicast based systems have difficulty expanding beyond a these boundaries. To remedy this, network designs may deploy application specific routers, or they shard the messaging system, or they use other protocols like mesh or gossip based systems. All of these solutions have advantages and drawbacks.

The aim of this system is to:

There are a lot of submodules and dependencies, so at present, building using the build Makefile is the easiest way to compile everything. Clone it, install the dependencies, clone all of the modules, build everything. The rpm dependencies will probably need the EPEL repo installed when using an enterprise RedHat, CentOS, or derivative for the liblzf-devel package (and maybe others).

If this completes, there will be a static binary at raims/OS/bin/ms_server where OS is something like RH8_x86_64.

If you set the env var for debugging, then the RH8_x86_64-g directory will be populated without optimization and with the -g flag.

The first task is to create the authentication keys for a service "test". The ms_gen_key program creates and updates the configuration. The user keys are what stored in the user_X_svc_test.yaml files and contain ECDH key pairs. The service is a ECDSA key pair and signs each user and stores the signatures in the svc_test.yaml file. The startup.yaml contains the startup config. The config.yaml file includes all of the files in the config directory.

This creates the keys for users A, B, and C. These keys are encrypted with the .pass and .salt files.

More about this in the [key config guide](keys.md).

Run the ms_server program and configure it. The -u option specifies the user and service. The -c option starts the command line interface, where the networks can be defined and connected. This following defines a mesh endpoint and saves it to the startup config.

The files are described in the [configuration] section and the transports are described in the Networking section. The authentication keys need to be distributed to all the nodes, but the networking config will be somewhat unique to each node.

The key configuration files are necessary to join the network. They authenticate peers and the message traffic that flows between peers. It does not authenticate the local bridging protocols RV, NATS, or Redis.

Generating a master config is done with the ms_gen_key program. The default location for the config directory is ./config, other locations are specified with the -d option.

Initially, the config directory is empty. Initialize with some users and a service name.

Exporting the keys for each of the nodes causes the .pass file the change and the unnecessary private keys to be removed. The only private key that remains, is for the peer. This trimmed configuration allows the peer to run, but not generate new peers because the private key of the service is not present.

Copy the A config to the A node/config, the B config directory to the B node/config, etc. The .pass file is unique for each peer so that it can be removed after running the server, rendering the configured keys unreadable until the .pass file is restored or the peer’s config is regenerated from the master config.

The copy of the master config includes a copy of the param.yaml, as that can contain global configuration, but doesn’t copy any local configuration such as startup and network configuration.

The master config will also work, so just copying it to the peers will allow them to run if this type of security is unnecessary.

The ms_gen_key option -o will concatenate the configuration into a single file:

Running ms_server -d config configuration from a directory and running ms_server -d test.yaml will load the configuration from a file. In both cases, the configuration loaded will be the same.

A test network can be set up using only the loopback interface by describing the network using a format output by the show_graph command. The format of this is:

The node line declares all of the users. The tcp_ lines describe how the users are connected. The number following the : is the cost of the transport.

Running the ms_test_adj program with this description will generate a configuration, saved in a file called "graph.txt" and output to "graph.yaml":

The -l option causes the links to be resolved by exchanging messages over the loopback interface. At the bottom of the "graph.yaml" created, there will be commands in comments to run this configuration. Running these will create 4 users in a network describe by the graph. The following uses those commands with the first three running in the background and the last with a console attached to it, but you could run each in a different terminal with consoles attached in order to test with the sub and trace commands to test how messages would be routed through the network.

In addition to "tcp" type links, you could also define "mesh" and "pgm" types, but the pgm would require non-loopback interface that has multicast, like a Linux bridge.

The above graph would create two meshes, with different costs for some of the paths. This would route messages over both meshes by sharding the subject space and using one mesh for half of the subject space and the other mesh for the other half.

There is a graphical interface to view the network using the cytoscape package.

Connect to the url http://127.0.0.1:8080/graph_nodes.html with a web browser and paste the graph text into the text box after erasing the existing text, then click "show graph".

The parameters section of the configuration is used to lookup values that can alter the behavior of the server. These fields can be set anywhere in the config files, but are usually in the "param.yaml" or "startup.yaml" files. Since the "config.yaml" includes "*.yaml", any yaml file in the config directory can contain parameters. Any field value pair which is not in a service, user, service, transport, or group section is added to the parameters section.

This configuration is a list of parameters:

The "parameters:" structure is optional and not necessary to define them.

The startup section can be used to start transports during initialization. This syntax is used by the save console command, but can also be edited. The following causes the transports named myweb to start with listen, then start mymesh and mytcp with connect. The listeners are always started before the connecters.

The hosts section can be used to assign address strings to names, similar to an /etc/hosts configuration. The values assigned to the names are substituted in any connect or listen configuration of a transport. For example, the following hosts are used in the connect and listen portions of the net transport.

A mesh type transport with connect uses both the listen and the connect addresses defined, since all peers can both connect and accept connections.

Two peers authenticate with each other by signing a message with a configured ECDSA key. This message includes a generated a ECDH public key. The ECDH key is used by each side to compute the secret using the corresponding ECDH private key. The secret along with a unique nonce, a time stamp, and a sequence number to create a temporary key that is used to encrypt a random session key.

For peers A and B to complete the key exchange, there are 4 messages:

If either AUTH fails the HMAC computation, then the authentication fails and one or both peers are ignored for a 20 seconds (or 2 times the heartbeat interval). It is possible that the latency of the key exchange is greater than 2 HELLO/HB messages so the nonce associated with the seqno/time pair is too old and the authentication must restart.

The ECDSA private key used to sign the authentication messages is either the configured key pair from the service or the configured key pair from the user. A configuration may not include the service private key in the case that a user has less privileges that the service, which has admin privileges. The service’s private key is able to sign users which don’t exist and are added to the system, but a user’s private key can only authenticate itself.

The ECDH algorithm used is EC25519. The ECDSA algorithm used is ED25519.

The following is from the Example Message Flow. This shows the HELLO/HB part of the key exchange, where peer A is ruby and peer B is dyna.

The seqno_2, time_8, cnonce_16, pubkey_32 are the A_seqno, A_time, A_nonce, and A_ECDH_pub. The user_hmac_16, start_8, and service ECDSA_pub are combined to create a hello_key which is used to authenticate the HELLO message stored in pk_digest_16, since the session key that is product of the key exchange is not yet known by dyna. The service ECDSA_pub is never sent over the wire so it is used as a pre-shared key in this instance. There is another pre-shared key used by the Key Derivation Function (KDF) to generate keys from secrets, nonces, seqnos, and time stamps. The KDF is seeded by a 640 byte salt and shared along with the service ECDSA_pub key in all of the peers that need to communicate.

The first AUTH message from peer B (dyna) to peer A (ruby):

The auth_seqno_8, auth_time_8 are the A_seqno, A_time values from ruby used to find the A_nonce (cnonce_16) in the HELLO message. These along with seqno_8, time_8, cnonce_16, and pubkey_32 are used to construct the temporary key to decrypt the auth_key_64, which is session key used by dyna in the HMAC computation that authenticates messages and compare the result to digest_16. The pk_sig_64 is the ECDSA signature of the message signed either by the service’s private key or by the user dyna’s private key.

After this succeeds, then ruby trusts messages from dyna that have a HMAC computation digest_16 included with each message, along with an seqno and time stamp to prevent replays.

The second AUTH message from peer A (ruby) to peer B (dyna):

The same exchange from the first AUTH message is used in order for dyna to trust ruby.

If a host is compromised and the KDF pre-shared key and service ECDSA_pub key are discovered along with a user ECDSA_pri key, then an unauthorized party could masquerade as that user.

One way to prevent this is to remove the pre-shared 640 byte salt file after starting a server or the unique password file used to encrypt the ECDSA keys in the configuration files. Both the salt and password are needed to decrypt the keys.

Another option is to use stdin for reading the configuration so that no secrets are stored in the filesystem. For example, this will configure ms_server through sending a configuration through ssh to a remote host:

The ms_server running on host will read the configuration from stdin (-d - argument) and then fork itself to run in the background (-b argument).

The function of the key exchange protocol is to initialize each peer with a random 32 byte session key. The function of this key is to authenticate messages. A HMAC calculation of the message is done by enveloping the message data with the key and hashing it using a AES based hash that results in a 8 byte digest:

Note that HMAC is traditionally performed as MD5( key.opad + MD5( key.ipad
message ) ) or SHA3( message + key ). The above AES construction is chosen purely for speed, since AES instructions are widely available and an order of magnitude faster than the other hashes. This may change in the near future with the addition SHA instructions.

The header of every message contains these 5 fields which identify the source of the message, the HMAC digest of the message, the subject, a seqno and a time stamp:

This header ensures that a message never contains the same bits and is always unique. It also allows the receivers to check that a replay has not occurred by tracking the sequences and time stamps for the subjects that it has seen previously. If the subject has never been seen before, then the time stamp is used to check that the message is at least as old as the last network convergence time stamp, described in more thoroughly in Message Loss. The bridge_16 identifies the source of the message and the digest_16 is computed with the source’s session key.

The subject schema used by the external bridged transports may introduce some incompatibilities when routing from one to another. The subscriptions and the patterns are separate operators internally. A subscription using wildcarding characters is allowed, but not interpreted differently that any other subject. A pattern subscription includes a field which causes the pattern to be evaluated with different syntax rules, Redis GLOB or NATS/RV. A publish is not interpreted as a wildcard, even when it contains wildcard syntax. Any string of bytes can be subscribed or published, but the wildcarding follows the syntax of the pattern type and uses a different subscription operator internally, as Redis does (sub, unsub, psub, punsub).

There is a special rule for subjects that begin with the prefix _INBOX., it is interpreted as a point to point message. This subject format finds the peers which are subscribers, typically just one, and sends the message point to point for each one. The subject and message are put into an envelope addressed for each peer. The peers that forward this message along the path to the recipient recognize this as using a different forwarding rule than normal multicast subjects. For example, the point to point rules for forwarding will use a UDP inbox protocol when OpenPGM is deployed. The point to point rule will still forward to all subscriptions of an inbox subject, but it is optimized for the case that there is only one subscription.

A Rai MS transports function is to join all of the peers connected through a node together in one virtual overlay network that provides basic pub/sub multicast.

A transport has two primary roles, the routing of messages between peers and the managing of protocol dependent subscription management and message framing. The internal transports (PGM, TCP Mesh) all use the internal protocol semantics for messaging. The external bridged transports (RV, NATS, Redis) have protocols with similarities, but they have unique behaviors that make them more complicated that the internal transports.

The design of the internal transports allow them to be used by any of the external transports, so RV can use a TCP mesh or PGM multicast or some of combination of them interconnected. Similar for NATS and Redis, they can also use PGM multicast as well as a TCP mesh. The routing of messages between peers is agnostic to the type of protocol that the endpoint clients are using. It is possible to use the Rai MS protocol directly as well. The ms_server console contains the ability to publish, subscribe without using an external client. The Telnet transport uses the console protocol. The Web transport serves builtin html pages that interface with the console protocol through websocket protocol.

There are two sides to transport configuration, the listener and the connector. Only the internal transports support the connecting side (PGM, mesh, TCP), the client side (RV, NATS, Redis, Telnet, Web) only uses listeners and do not have a cost. The device option will auto-discover a connector or listener via multicast through a device. This requires that the connector and listener are on the same broadcast domain or have multicast routing configured.

The config file format is a JSON or YAML with a record that can have these fields:

The name identifies the transport so that it can be referenced for starting and stopping in the console and the command line. It is also used by auto discovery to match transports and it is sent to other peers so that it can be read in log files and diagnostic output. It has no protocol implications beyond auto discovery, a misspelling won’t cause it to stop working.

The endpoint protocols: RV, NATS, Redis; all have a service defined to separate the data flow from one another. By using the same service name allows these endpoints to share the same namespace. The underlay network that connects the namespaces can both be configured using the YAML files or the console and also be specified by the connecting clients. The clients can specify a network with PGM multicast or with TCP endpoints and meshes. All of networks specified by a client that use TCP will still use multicast to resolve the endpoints by service name by using the name protocol.

Networks use a device name and a protocol or a multicast address. When a network is not specified by a client or configuration, then the links between services have to be configured by the YAML files and/or in the console.

Example networks and how they are interpreted. All of these have a service name associated with the network, which must match for namespace to communicate.

The device name eth0 can be substituted with an IPv4 address, like 192.168.1.0;tcp.listen, or a hostname that resolves to an IPv4 address. If a network is specified without a name, like ;tcp.listen, then the machine’s hostname is used to find the device.

The configuration for the PGM, name, TCP protocols are generated as needed by the client if they do not exist. When a service already is configured, then it is used instead and the network parameters are ignored.

All of the internal transports have a cost assigned to the links. The routing from peer to peer uses this cost to find a path that minimizes the cost. Equal cost links are utilized by each peer by encoding a path into the message header. This path is enumerated from 0 → 3, so there is a maximum of 4 equal cost paths possible between any 2 peers in the network. The per path cost can be configured by using different cost metrics for each link. The default cost is 1000 so that a configured cost can be less or greater than 1000. These configured metrics are replicated throughout the network so that every peer agrees the cost of every path that exists. A case where lowering the cost is useful is when some of the links have higher performance than others as is the case when all peers exist within a host or within a data center. A case when configuring different cost for each of the 4 paths is to load balance multiple links with equal performance.

Example of configuring a lower cost mesh on a bridge:

If every container within this host has a RV client that connects with a network and service of -network eth0;mesh -service 7500 then the cost of 10 discovered through the docker0 bridge. The name protocols used will use the name of the device as their tport name.

Example of configuring a load balanced cost for links through a data center:

This creates 4 equal mesh networks, each of which is preferred for part of the subject space. The connect and cost can be enumerated as connect, connect2, connect3, connect4 and cost, cost2, cost3, cost4 as well as an array.

The TCP type and mesh type links are encrypted using AES 128 bit in counter mode. The protocol above the link layer handles the authentication for trusting the peer and the messages that are received, described in Authentication. The encryption is set up by a ECDH exchange. Each side generates a ECDH keypair and sends the public key with a checksum and a 128 bit nonce value. Each side computes the secret key and uses the KDF to mix the secret with the nonce value to arrive at a 128 bit key and a 128 bit counter for sending and receiving. Thse are used to encrypt and decrypt the other sides bytes.

The 32 byte secret will be the same on both ends. The nonce is a random 16 byte value. The KDF function mixes into the keys a preshared salt value, generated by ms_key_gen in a "config/.salt" file described in Configuration. Without this salt value, the key exchange will compute incorrect keys even though the secret is computed correctly.

The 8 bytes checksum is a CRC of the pub key and the nonce in big endian, so the first 4 bytes will be zero. The zero bytes cause an encrypted connection to an unencrypted endpoint to fail.

The 64 byte result of the KDF computation is folded with XOR to arrive at the 16 byte AES key and the 16 byte counter value.

PGM is a multicast protocol, which layers reliability on the native UDP multicast. The parameters for it declare the amount of memory used for buffering data and control the timers when retransmitting is necessary.

The type of PGM used is UDP encapsulated using the port specified. The address specification has a network, a send address, and multiple receive addresses, formatted as network;recv1,..;send, so this is a valid address: 192.168.1.0;224.4.4.4,225.5.5.5;226.6.6.6 where the send address is the last part and the middle addresses are where packets are received. If the network part is unspecified, then the hostname is used to find the interface. If there is only one multicast address, then that is used for both sending and receiving.

Example tport_mypgm.yaml:

The transmit and receive window sizes expand to the reliability time or the txw_secs parameter. When the txw_secs is not set, then the reliability passed on the command line or as a configuration parameter is used. The receive window memory is not used until there is packet loss and a retransmission occurs. Unrecoverable packet loss occurs when the transmission window no longer has the sequences that are lost. The mcast_loop, when set to 2, allows two peers to share the same network on the same host. This causes packets to loop back through the interface and allows multiple PGM networks to coexist on the same multicast group.

In addition to the multicast networking, an inbox protocol is used for point to point messages. The network specified in the multicast address is used as the inbox network, with a random port.

The listen and connect addresses act similarly, two peers using different methods will communicate if the multicast send address matches one of the receive addresses and the inboxes are connected.

A TCP mesh is a group of peers which automatically maintain connections with every other peer. When a new peer joins the mesh, it opens a connection with all the other peers which are currently members of the mesh.

The timeout parameter causes the connecting peer to retry for this amount of time. When the timeout expires, the transport will not try to connect until told to do so again.

Multiple connect addresses are normally specified so that some connection likely succeeds if that network is running. Allow peers can specify multiple connect addresses since they use both listen and connect methods to join a network. After one connection succeeds, all the other connections in progress are stopped and the list of mesh members are downloaded from the peers and those are connected.

Example tport_mymesh.yaml:

If the mesh is a stable network, then the timeout set to a larger value or zero can prevent a network split where some parts of the network are isolated for a period of time. When a host is restarted doesn’t have as much of an effect by a timeout since it is rejoining an existing network. If a timeout expires, then an admin request to rejoin the network is possible through the console.

When a device parameter is used, then multicast is used through the name protocol to discover peers that are joining the same mesh, matching using the tport name. After discovering the peer, a connection with TCP is used to join the mesh. The port can be random with a device, since the address is discovered rather than connected. Both the device and connect can be methods can be used.

The noencrypt parameter set to true disables tcp link encryption. Both the listener and connector must match this setting, otherwise they will close the connection after receiving the first bytes sent.

A TCP point to point connection to another peer. This is useful to create ad-hoc topologies at the network boundaries.

Example tport_mytcp.yaml:

A TCP protocol is either a listener or a connector, the appropriate config is used at run time when a connect or listen is used to activate the port. When device is used to discover the peers through the multicast name protocol, the listeners are matched with the connectors. When more than one listener is discovered by a connector, then connections are made to each one.

Whether a configuration is used to connect or listen is specified by a listen or connect command or configuration. If multiple connections are specified by the connect parameter, then the first connection that is successful will cause the others to stop trying to connect.

The edge parameter set to true causes the passive peer to pool the connections on a single transport, similar to a multicast transport where the traffic is primarily through a gateway peer. The noencrypt parameter set to true disables tcp link encryption. Both the listener and connector must match this setting, otherwise they will close the connection after receiving the first bytes sent.

If the listen or connect parameters specify a port, as in "localhost:8000", then that port overides the parameter port configured. A device name is resolved before the hostname DNS resolver is tried, so "eth0:8000" will resolve the address configured on the eth0 device.

The RV protocol supports both the RV5 and RV6+ styles of clients. The RV6+ clients use the daemon for the inbox endpoint and don’t create sessions, the RV5 clients use a unique session for each connection and allow an inbox reply in the subscription start. These differences cause decades old software incompatibilities and pressure to re-engineer legacy messaging systems.

There clients usually specify the network and service they want to connect, which is different from the other clients. When a client requests to connect to a multicast network, the Rai MS ms_server will start a PGM transport for it, unless an existing transport is already defined named with a rv_ prefix and a service numbered suffix.

When the rv_7500 transport exists as a TCP mesh, then this network is remapped to the predefined transport when a RV client uses the service 7500 and the multicast network specified by the client is ignored. When no multicast network is specified, then no Rai MS transport is created and the existing transports are used.

Example tport_myrv.yaml:

Unless the use_service_prefix is false, the traffic is segregated to the _rv_7500 where service is 7500. If it is true, then all services that also have use_service_prefix set to true will share the same namespace. Without no_fakeip set to true, the session and inbox values are random and not based on the IPv4 address of the host. This allows RV networks to work without a routable IPv4 network across private address spaces that are common with NATs, VMs, and/or container networks.

NATS is a pub/sub system that is similar to RV with respect to subject schema with some extensions for queue groups and optionally persistent message streaming. The protocol support does not include the streaming components, only the pub/sub and queue groups. NATS does not have an inbox point-to-point publish scheme, it relies on the client to create a unique subject for this functionality.

Example tport_mynats.yaml:

If the network is specified, then starting the NATS service will also join the network. A network format is as described in Services and Networks.

Redis has a pub/sub component that has slightly different semantics, without a reply subject for request/reply. It also uses the term channel to refer to a subscription. A pattern subscription is separated by a psub operator, allowing subscriptions and publishes to any series of bytes.

Example tport_myredis.yaml:

The data operators that operate on cached structures like lists and sets, etc. These commands are available when a shared memory key value segment created and passed as a command line argument to the server (example: -m sysv:raikv.shm), or defined as a value in the config files (example: map: "sysv:raikv.shm").

If the network is specified, then starting the Redis service will also join the network. A network format is as described in Services and Networks.

Telnet is a way to get a console prompt, but it doesn’t start by default. It uses the same transport config as the pub/sub protocols. It also can be used by network configuration tools to install a configuration remotely. A telnet client signals the service that it has terminal capabilities which enables command line editing.

Example tport_mytelnet.yaml:

Web handles http requests and websocket endpoints and integrates an web application that can be used to graph activity and show internal tables. The web application is compiled into the server, so no external file access is necessary.

Example tport_myweb.yaml:

If http_dir is not set, then this service does not access the filesystem for processing http get requests. It has a set of html pages compiled into the binary that it uses for viewing the server state.

If http_dir is set, then the files located in the directory will override the internal files. The html files and websocket requests also have a templating system which substitute values from the server. If @(show ports) appears in a html page, it is replace with a html <table> of ports. If template "res" : @{show ports} is sent using a websocket, it expands to a JSON array off ports and the reply is "res" : [ports...].

Any of the commands from the console interface are now exposed on the http endpoint. Requesting "show ports" will respond with a JSON array of transports with the current totals of messages and bytes:

The websocket endpoint can also be used to subscribe subjects. When a message is published to the websocket, the format used is:

This requires that the messages published can be converted to JSON or is already in JSON format.

The http_username / http_password or htdigest will cause http digest authentication to be used and require them for access. The above wget is used with the example configuration.

A htdigest file contains a list of users and can be created by the htdigest program distributed with the Apache packages.

Then tport_myweb.yaml is configured like this:

Only one realm can be used by the service. If http_realm is configured then that realm is used, otherwise the first realm in the htdigest file is used. If no realm is specified but a user and password are specified, then "realm@raims" is used.

Each node in a network must construct a forwarding set for any message sent by any peer. A forwarding set instructs the node where to send a message so that all subscribers of it will see the message exactly one time, when the network is converged and stable.

A "converged network" is one where all peers agree that a link exists. If peer A has in it’s database a link to peer B, then peer B must also have a link to peer A. If a link is missing, then the network tries to resolve the difference by asking the peers with the discrepancy which is correct.

Every peer has a bloom filter that contains all of the subscriptions currently active. The links database tells each peer how the network can be traversed for full coverage and the bloom filter prunes the coverage by dropping the message when there are no subscriptions active that match the subject on the other side of the link.

A simple redundant network is a circle:

If the cost of each of the links is set to the default 1000, then the forwarding set for dyna is the link to ruby and bond. When ruby and bond receive a message from dyna, only one of them will forward the message to chex. The path cost from dyna → ruby → chex is equal to the path cost from dyna → bond → chex. The forwarding algorithm tracks the equal cost paths and ranks them in order of peer age. In the case that ruby is older than bond, then the ranking of these routes would by 1. dyna → ruby → chex and 2. dyna → bond → chex. The top 4 ranked routes are saved as the forwarding sets, and selected by the hash of the message subject. In this case, half of the subjects subscribed by chex and published from dyna would take the first path and the other half would take the second path.

The method of ranking the paths by peer age is used because the stability of the network is less affected when more transient peers are added and subtracted from the link state database.

The sequence numbers include a time frame when the publisher starts the message stream. This is the computation that creates a new sequence stream.

This truncates nanosecond time stamp to approximately 10 second intervals, a new time frame can only occur after 10 seconds. The time frame is stored in the upper 29 bits will be valid until the year 2115. The sequence resolution within a time frame is 35 bits or 34 billion sequences. These are rotated to new time frames when the sequence number is zero.

These are properties of the time frame encoded in the message sequence numbers:

All of the transports are stream oriented, so a loss of unrecoverable network packets will cause connections to drop and a new convergence state by pruning the lost routes. All peers will agree on a time that convergence is reached. New time frames are created for all messages published so that the time frame constructed in any one peer greater than the convergence time in all peers.

When routes are added to or subtracted from the network, the message routing is not stable until all peers have finished adjusting their view of the network. The peer that publishes a message may use a sub-optimal forwarding path to the recipients until they are notified that better paths are available with link state exchanges.

A map of subject to sequence numbers for published multicast messages is maintained by each peer. This map rotates when a configured memory limit is reached, pub_window_size, and the window time interval is greater than a configured time, pub_window_time, which must be at least 10 seconds. When a subject is rotated out of the window, the sequence number is restarted with a new time frame.

A map of subject to sequence numbers for the subscriptions that a peer has is also maintained. This validates that the messages are processed in order and allows notification of message loss when the sequences skip and does not allow a message to be processed twice. The memory limit for this is sub_window_size and time limit is sub_window_time. When a subject is rotated out of the window, then the publisher did not update for the window time and the next sequence is treated as if a new subscription was created.

Message duplicates are avoided by discarding messages that are older than the trailing edge of the subscription sequence window. The clock skew between systems is estimated. The console command show skew will display the calculated clock skew between systems.

The pong calculation subtracts the round trip time and is the most accurate, the others disregard the latency of the network. The HB are from time differences of directly attached peers using heartbeats and are shared with those not directly attached. The ref is the peer (0 = self, 1 = A.1) that originated the HB difference. The time is the estimated clock setting of the remote peer in the current timezone.

The sizes and windows are in the parameters section of the config file and default to 4 megabyte (about 60,000 subjects for publishers and 20,000 for subscribers) and 10 seconds. The size of the windows will have an overhead of 48 bytes for publishers and 128 bytes for subscribers in addition to the subject size. The 10 second rotate timer could cause more memory to be used if lots of new subjects are published or lots of new subjects are subscribed within 10 seconds.

The show loss console command displays the messaging statistics for the peers.

An inbox message loss is not unusual since the point to point messages are often used for link state exchanges and other network convergence functions. Inbox message loss is usually not as problematic as multicast message loss since there often timers are retries associated with their usage.

Multicast message loss is much more difficult to recover from, since there are usually many multicast streams and tracking the state of each one is a problem solved by persistent message queues. This requires clients which track the state of the messages they consume and notify the queue when they are finished with processing them.

If a message arrives with a sequence which is not in order, it is forwarded with state indicating how many messages are missing, if that can be determined. The protocol handling of this notification is to publish a message indicating how many messages were lost.

Each field in a message is encoded with a type and length. This allows new fields to be added without disrupting the message parsing. The first 16 bits encodes the type, length and field id. The rest of the field encodes the value. All integers are encoded in big endian.

The types defined are bool (size:1), unsigned int (size:2,4,8), opaque (size:16,32,64), string (max size:64k), long opaque (max size:4G).

The first two bits, fixed and primitive, indicate whether the type has a fixed length, and whether the value is a field (primitive) or a message (not primitive). A message is another group of fields and is always encoded as a long opaque with the primitive bit set to 0. A message payload is always encoded as a long opaque with the primitive bit set to 1.

The types are enumerated as:

The field values are aligned on 2 byte boundaries, so the value is padded one byte when the value size is odd. There are currently 76 different field ids (fid) and a maximum of 256 (defined in the header file msg.h).

A message frame has 5 fixed length sections and 3 fields that are always present and use two bytes.

These header fields are:

The first 4 bytes encoded as:

The Message Type encodes 4 classes of messages:

A message that has routeable data always has the Multicast or Inbox type set. The Inbox type message is also used for RPC style communication between peers. The Router Alert type message alters the routing database by modifying the link state or the subscription state. A Heartbeat type is a periodic presence update. The peers which are directly connected are responsible for detecting link failures.

The Option Flags is a bit mask that encodes options for messages with Multicast and Inbox types that are routing data payloads to endpoints. These are:

The message size does not include the first 8 bytes, so the message frame size is 8 + the message size field. If the size is greater than 24 bits, then the next 32 bits are used to encode the size and the subject hash is calculated from the subject.

The Bridge, Message Digest and Subject are encoded in Type Length Value format. The Bridge is a 128 bit identity of the sender. The Message Digest is the authentication field. The receiving peer will authenticate that the message is valid by using the Bridge to look up the 512 bit session key of the sender and calculate an HMAC using the message data with the session key and compare it to the value contained in the Message Digest. In addition, there are sequence numbers and time stamps present that prevent the replay of each message frame.

The 4 multicast path options will select the one of the equal cost paths calculated from the current link state. Every peer can calculate these paths using the same replicated link state database, this results in 4 forwarding trees to the same destinations if there are enough redundant links.

The peers exchange messages to authenticate new peers, synchronize the link state of the network, subscription updates, and heartbeats to maintain neighbor links. These types of messages have unique subject prefixes as well as bits in the message type header indicating whether it is special.

There are 7 classes subject prefixes used:

A broadcast style forwarding used by _Z, subjects is different from multicast forwarding. It will flood the authenticated peers in the network, adjusting each peer’s routing database as it is received. It uses this type of forwarding because this kind of update may cause the multicast forwarding to be temporarily incomplete until the network converges again.

The forwarding path for the Inbox, Heartbeat and broadcast subjects does not follow the multicast forwarding path, so they can’t be subscribed.

There is a separate sequence number domain defined for these because of the idempotent nature of maintaining the replicated state of the network. If a peer misses messages for a delta changes in the subscriptions or links database, the state is reinitialized by replicating it from an up to date peer.

The multicast subjects follow normal forwarding rules. The _M prefix is used for a multicast ping and a multicast link state sync.

The _N prefix has unique subjects for link and peer statistics like messages sent or received, bytes sent or received, as well as adjacency notifications. These are used to monitor an individual node or a group of them with pattern subscriptions. These stats are not sent unless there are subscriptions open.

These are sent on a link between directly connected peers.

These are broadcast flooded to authenticated peers.

These are multicast to authenticated peers. They are updates to the bloom filter that can be missed and resynchronized with _Z.BLM or a resync RPC request.

The format of a subject with an _I. prefix also encodes the destination of the message by appending the 128 bridge id in base64.

Example:

All of the peers along the path to the destination use this bridge id to forward the message using the rules for the point to point route of the destination peer. This may be a TCP link or it may be a UDP Inbox link in the case of a multicast PGM transport. The suffix of the inbox subject indicate the type of request or reply it is. If the suffix is an integer then the endpoint is not a system function, but information requested by the console session or a web interface that is usually converted to text and displayed.

These suffixes are currently recognized:

Two peers key exchange, ruby connecting to dyna:

Ruby connecting dyna, a member of a network of 4 nodes: dyna, zero, one, and two. This is the message flow between ruby and dyna, which completes the initial synchronization of ruby.

There is also message flow between dyna and zero, one, two. This is the flow between dyna and zero. The message flow between dyna and one, dyna and two is the same as dyna and zero.

Adding ruby to the network ripples through the directly connected peers, which discover the new peer from the broadcasting of the _Z.ADD messages and then synchronize with each other to merge the ruby session key, the link state, and the subscription bloom state into the network state.

If ms_server is started in rvd compatible mode, it will automatically start a rv protocol on 7500 and a web service on 7580 unless arguments are present that modify this. The protocol that is used between daemons is not compatible with rvd, but it does allow rv clients to connect and communicate. In other words, the client side is compatible, but the network side is not.

These arguments are recognized:

Without any arguments, the config file rv.yaml is loaded from the directory that ms_server is installed. This config file can be generated with the ms_gen_key program. It should be the same for each instance that is joining the same network and service, since it contains the service key pair that authenticates the daemon with other daemons located on the network.

If ms_server is installed in /usr/local/bin then this can generate the default config file for it in rvd mode:

The /usr/local/bin/rv.yaml file must be installed on every machine that connects to the network and expects to communicate with the initial machine. The contents define the service key pair:

The above service key pair is unique for every ms_gen_key execution. The private key is used to sign the authentication messages exchanged between daemons, and the public key is used to verify that the peer is allowed to exchange messages on the network. Unauthenticated peers will be ignored.

If the ms_server is linked to rvd and run that way, it will run in compatibility mode:

Unless the -foreground or the -console options are used, it forks itself to release from the terminal that it is started. ms_server will also run in compatibility mode when an argument above is used, for example, ms_server -listen 7501 -http 7581 -reliability 60 will run in compatible mode.

If there is already a rvd running on port 7500, it will fail to start and exit:

A HUP signal will cause it to exit:

The network parameter that the client specifies controls which network that the ms_server joins. It can specify a multicast address, TCP connections, or a TCP mesh. Only daemons which connect to the same network will communicate.

The formats of these are:

A mesh network causes all the daemons to connect with one another by listening to a random port and multicasting that port to eth0. When other daemons receive this message, they will establish TCP connections with each other daemon.

A TCP network causes the listeners to multicast their random ports to eth0. When daemons that have tcp.connect as a network receive this message, they will connect to the listener. Multiple TCP listeners can exist on the same network. The result of having two "eth0;tcp.listen" specifications and two "eth0;tcp.connect" would be that both connectors will establish connections to both of the listeners.

The PGM multicast address uses UDP encapsulated multicast on the service port using OpenPGM and a UDP point to point protocol for inbox messaging.

The sockets will be bound to the eth0 interface with random ports, except for the PGM socket, which uses a wildcard address for joining the multicast and the service port for sending messages. Multiple services can join the same network, so -service 7500 and -service 7600 can coexist using the same network specification.

When two ms_server instances are using the network "eth0;mesh" on service 7500 and service 7600, the ports console command will show these networks:

The ANY specifier can either connect to a mesh or a TCP listener, depending which is present.

The empty network does not attempt to connect to anything, but it will find other services through existing connections.

If there exists a rv_7500 transport in the configuration (configured in rv.yaml or the -cfg argument), this overrides any client specified network connection for service 7500, so the client network argument is ignored.

Each ms_server instance uses the hostname of the machine to identify itself unless the -user argument is used to specify another name. The daemon port is appended to the user name so that multiple daemons appear as hostname_7500 and hostname_7600 when -listen 7500 and -listen 7600 are used for two different daemon instances.

The console of ms_server is available when the -c option is used or when a Telnet protocol is defined. It offers command line editing and completions. It can be used to define, start, or stop connections between instances and also modify which IPC protocols are running for clients to use. It also has many ways to examine and debug the network.

The output is usually colorized if the terminal supports it, with green and red used for log messages (normal and error) and white used for cli command execution results. Printing messages received are also colorized, green for field name, blue for field type, white for field values.

The user names and the transport names usually have an integer number appended to them, for example lex_a2.3 is the user lex_a2 that has a uid of 3. This indicates either the uid or the tport_id of the identifier. The string identifiers of users and transports can contain duplicates, since they are identified using the bridge id. The bridge id is a unique random 128 bit nonce, the strings attached to the users and transports are tags which usually are unique, but not necessarily. The users and transports are kept in their respective a tables and the uid and tport_id are indexes into these tables. The * is often used for uid 0 so that it stands out, since it is the peer that the console is attached to. The tport_id of 0 is also special, that is where the client protocols are attached through local IPC, for example, a TCP connection to 127.0.0.1:7500.

The command string entered into the cli will execute if it has enough characters to distinguish it from the prefixes of other commands. If the string pi is entered, then the command ping will run, since pi is a unique prefix of ping. The show prefix is optional when the command matches the second part of the show command, so pe will match and run the show peers command. The shortened command run t test will match and run the show running transport test command.

The following is the help screen, displayed when "help" is entered at the cli.

The arguments in square brackets are optional, the letters used above are:

These commands send a message to a peer and display the message returned. The tping command also sets the trace flag in the message sent so that all peers along the path will also send a message back. This is useful in the way that traceroute is useful, to find an unusual latency report or dropped messages.

The ping and tping optionally have an argument that specifies the name of the peer to send the message. If no argument is used, then every peer currently active will be sent a message. These messages are sent over the link that is handling the inbox point to point messages. The subject of a ping message uses the inbox format _I.<nonce>.ping, where the nonce identifies the destination peer. The return uses the _I.<nonce>.N inbox subject, where nonce identifies the peer of the sending console. The N part of the subject is setup by the console to identify what the sending operation was and is used in the reply field of the original message.

The mping use a multicast path instead of an inbox path. The multicast path is numbered and is added to the message header so that all peers which receive and route this message will use the same path. All peers that receive it will send an inbox reply message, similar to ping. The subject used by the sender is _M.ping, which all peers are subscribed to. The multicast paths are numbered 0 to 3, so mping 0 will use the first path, and mping 3 will use the last path. Using different paths can be useful to check that all redundant links in use are active and forwarding. The reply also includes which port the message was received on, which will match the path 3 network path. The path 0 is often the same as the inbox path, except in the case of PGM, where inbox is a UDP point to point protocol.

If the network is not yet stable, sometimes a ping operation will not complete. When this occurs, use the cancel command to show the completed and the incomplete values. When a ping operation is started, the console estimates the number of replies that are expected and waits for these to complete before displaying the results. The tping will display the acks of the message as they are received but wait for the final results.

Example ping.

Example mping.

The tport field is where the reply inbox message was received, the peer_tport is where the ping message was received at the peer.

Remote will message a command to another peer, run it in it’s console and return the result. This is useful because most often, a peer will not have a console, a web interface, or a telnet protocol active. Without remote, the peer would need to be restarted in order to change the configuration or start a console. With remote, you could connect a peer with authentication, encryption and a console to the network temporarily, make a change, then disconnect the peer.

Example of remote.

These commands show and modify the running configuration. The save command write the running config to the startup config, when the directory and files are writable.

The show running and show startup will print the config tree in yaml to the console. The running configuration may have some dynamically created users and protocols which are created as a result of the startup config. A dynamically created user that is not preconfigured is one of these. These will show in running, but will not save to startup.

Using the configure transport command is the most often used command of these. It will update the currently running transports as well as add new ones. If it is used to modify an existing transport that is already running, the new settings won’t change the active transport until it is restarted with shutdown and connect or listen. The configuration details of transports are described in Networking, and the details of the parameters are described in Parameters. Most of the parameters are only applied at startup, so changing them will have an effect only when saved and the process restarted.

Example of configure transport and show running transport.

The first configure command enters into a cli sub command mode where only the fields of the transport can be entered. The second configure command sets all of the fields on one line.

The commands show service and show group have limited usefulness at in the current implementation, since only one service is used per ms_server instance and groups do not have operational functionality yet, eventually they will be used for access control lists.

The transport T is defined before using the connect, listen, shutdown commands. The network command configures the transport if not already configured, runs it, and also attaches a service to it. The configuration of the transports is described in Networking.

Example of connect, listen, shutdown.

Show the subscriptions active for user or for all users. The W is a substring for partial matches. This command uses inbox RPC calls to _I.<nonce>.subs for all users which U specifies. The * user matches all users, so the W argument can be specified.

Example, show all subscriptions for every user:

The (p) strings before the subject indicates that the subject was subscribed as a pattern.

Example, show all subscriptions which have the substring DAEMON:

Example, show subscriptions active at user edo_a3:

Show the sequences of the subjects received and published. The peers with IPC or console subscribers or publishers track the sequences the subjects to ensure the stream is completely serialized and notify of a data loss error when it is not in sequence. The details of how this works is described in Message Loss. This command only operates on the local sequence windows, the show windows command shows the memory usage of these.

The W is a substring that matches the subject so that the subjects in the window can be filtered. Without W, all of the subjects are printed.

Example, show the sequences of the subjects which contain ORCL:

The source is the publisher, so IPC indicates that the client attached to the lex_a1 has published these messages, and dex_a2, dex_a1 indicate that these messages were received from clients attached to those peers (or the console). The start is the first time in the time frame that the subject was seen, the time is the last time it was seen. New time frames occur when the network link state database changes, since the sequence number time frame reference jump between old and new time frames and the seqno base is linear.

Show the adjacency tables. This command dumps the current link state database. It shows which peer has a link to another peer through which tport and the cost of the link (of path 0).

Example:

The user is the peer that is maintaining the links that follow. It sends a link state update messages when a link is added, dropped or cost is changed.

The adj field is the peer which is directly attached to user through the tport. The tport is the name that user is labeling this link. The tport_id number that follows the name (fo_mesh + .4) is the index into the user’s transport table. The type and cost fields are also sent by user in the link state update.

Shows info about the peers in the network that are active.

Example:

The bridge is the 128 bit random nonce created on startup by each peer. It uniquely identifies the peer instance.

The sub field are the number of subscriptions that are active. This number is a counter in the bloom filter that is updated by the peer when subjects and patterns are added or removed. It always contains at least 2 entries, one for the _I.<nonce>.> inbox pattern and one for the _M.> multicast pattern.

The seq field is the sequence number for each subscription operation. It is serialized so that all subscriptions happen in the same order as the peer.

The link field is the sequence number for each link state update. It is also serialized so that adjacency table modifications occur in order.

The lat, max, avg are ping round trip times that are sent 1.5x the heartbeat interval to a random peer. They are tracked for at least an hour before being rotated.

The time is the start time of the peer.

The tport and cost reference the inbox route to peer.

The order in the table is by uid. Using the show peers nonce orders the table by bridge nonce, show peers start orders the table by start time, show peers user orders the table by user name. The show peers host shows the first 4 bytes of the bridge used as the host id and show peers ip shows the first 4 bytes of the bridge in IPv4 dotted quad format.

Using show peers zombie, the dead peers are displayed.

Show info about transports that are active on the network.

Example:

The tport, type are configured, and the cost is either configured or advertised by the peer in it’s link state message. If a transport is internal, like an IPC transport, then it doesn’t have a cost associated with it.

The fd field is the endpoint for the transport, usually a listener or a fd assigned to the transport. There are usually one or more fds within the transport that carry out the reading and writing of data to a network endpoint.

The fields bs, br, ms, and mr fields are bytes, messages sent and received, which are collected from all the fds within the transport.

The idle is the last time a message event occurred.

The fl field are flags that are set on the transport. Each character is a different flag:

The address field is the address at the peer when TCP is used and the multicast address when PGM is used.

This is similar to show ports except that all 4 costs are printed for each transport.

Example:

Similar to show ports with a status errno if the system reported an error on a link. When everything is normal, the address is printed instead.

Example:

Show the routes. This shows how all the peers are connected and which port would be used to send and receive messages to/from the peer. It also displays which transports have been used in order to reach the peer.

Example:

This shows that user pic_a2 messages have been received or sent through these transports. The secondary transports are often used on startup when the other links are not yet active or when a link fails.

The state of the transport has these values:

The cost is the link cost of the path P argument, or 0 when not specified.

The path field enumerates which transport is used to reach peer for each path.

The lat, fd are the same as show ports.

The route is the directly connected peer address that a message is sent or received.

Show the local and peer addresses as well as the url used to resolve the address of the peer. This is useful for mesh and multicast type networks since the endpoints are sometimes resolved through exchanging messages with the network. In the case of a mesh transport, a mesh url database is exchanged and links are established with all the peers that are in the mesh. The multicast PGM transport exchanges the unicast UDP endpoints for all the peers that are on the transport.

Example:

The top section is similar to show ports with addition of the urls.

The following sections is similar to show routes with the addition of the urls for each user.

The url field is resolved by exchanging messages. The local and remote are addresses assigned to the connection. Since a mesh may be actively connected by either peer, since all peers passive listeners and some have active connections. The newer peers will usually have the active connections and the older peers will have accepted connections. The local and remote addresses will reflect that, since the accepted peers are assigned an address by the system and the connecting peers use the url address to connect.

Show the state of the transports. This prints the configured transport and whether it is active or not. The other transport show commands will only show the active transports. This will show the ones configured but not active as well.

Example:

The listen, connect, and device fields show how the transport is configured to resolve the connections.

Show the users configured.

Example:

The system tracks the authentication and transport and link state events in a buffer that rotates every 4096 entries. This is a compact table that has 6 integer fields that map to a time stamp, uids, transports and enumerated values depending on event type. These events are useful for resolving what happened to the network after something went wrong.

Example of an event log:

The events that are logged are:

The last log 64K bytes of the log is buffered in the process. This command shows the this buffer.

Show the counters of heartbeat, inbox, and ping subjects.

Example:

The start field is when the process started. The hb seqno and hb time track the last heartbeat received from the peer when it is directly connected. The snd ibx, rcv ibx are counters for many of the _I.<nonce>. subjects which guard against repeats. These are point to point messages, the peer has the same counters which should match these. The show inbox command will show the last 32 of these sequences. The ping and pong sequences have their own counters, since these are used to check connectivity between peers and are expected to have loss when the network is unstable.

Show the link state seqno and sub seqno sums.

Example:

The start field is when the process started. The link_seqno and link_sum are the link state seqno and the sum of all of the peers link state seqnos. The sub_seqno and sub_sum are the subscription seqno and the sum of all peers subscription seqnos. These sums will only appear then the nodes is directly connected to the peer, since they are the values last seen in the heartbeat messages.

The sequence numbers are always increasing after a change in the link state or subscription state, so the sums of these seqnos are unique for the current network state and provide a way for peers to check whether they are in sync with the network.

These are exchanged with the heartbeat messages. When a difference is detected, the hb_diff is incremented and a _M.sync message is multicast to the network. When a peer receives the sync message, it checks that their sums match with the sending peer. If they do not match, then they reply with their current link state and subscription seqno values in a _I.<nonce>.sync point to point message. When a peer receives the sync reply it checks that these are in sync and requests adjacency with _I.<nonce>.sync_req if they do not.

The hb_diff may not always result in an actual difference with the network, since it is possible that a subscription or a link state message is received and applied to the peer at a different rate than the heartbeat is received, but the reply of the current sequence numbers at the peer will most likely be less than or equal the state of the network when the peer is in sync.

The mc_req is the number of _M.sync message received, mc_res is the number of _I.<nonce>.sync messages received. The req_adj is the number of adjacency requests made as a result of the _M.sync messages, and res_adj is adjacency requests made as a result of the _I.<nonce>.sync messages and ping_adj is adjacency requests made as a result of _I.<nonce>.ping messages.

When a message header is created or unpacked, a counter of the subject class is incremented. This shows these counters. These are only messages that are processed by the network, it is possible that two clients within the IPC transport are exchanging messages, these are not counted.

Example:

Show the types of the last 32 system RPC messages sent and received for each peer. Some peers may not have any of these if they are not directly connected.

This is an example of a peer attached to the console connecting to a larger network:

The first 3 sequences are the result of authentication, which causes both peers to exchange all their known peers. The following u_inbox_sync_req and u_inbox_sync_rpy pairs are used to request the peers which are not yet authenticated. In this case, the connecting peer has no peers and the peer attached to the network has lots of peers that need synchronizing.

Show the counters of repeated messages (old message sequences), messages not subscribed, have message loss, or have inbox loss.

When a message is repeated or not subscribed, a counter is incremented and the message is tossed. These types of events can occur through normal operation and don’t have an impact on clients.

The repeated messages can occur during network instability and not subscribed messages can occur because an unsubscribe has not yet reached the publisher or because the bloom filter did not filter the subject.

The message loss counters are more critical to correct behavior, since this indicates that messages did not reach all subscriptions. The inbox message loss can occur normally since these are used to synchronize peers during network instability, they are used to stabilize the network.

The point to point messages using the _INBOX prefix will also use the inbox sequences, but even these are not as critical since clients will have timeouts and retry the operation that uses an _INBOX

Example:

The user is the sender of the message. The repeat, rep time is the count and time stamp of the last instance. The not sub, not time are for the not subscribed messages. The msg loss, loss time are for the multicast message loss. The ibx loss, ibx time are for the point to point inbox message loss.

Show the system time skew between peers. There are several messages that include a time stamp which can be used to estimate the system clock skew between peers. This is useful to guard against message replays. If a peer message arrives and the time + skew is older than the subscription window, then it is treated as a repeated message. When the time is within the subscription window, then a sequence will be associated with the last message received from peer. The subscription window rotate time is configurable, described in Parameters of the config section. This details of the loss calculation is described in [msg_loss].

Example:

The first message a peer will see when connecting is the heartbeat message and authentication messages. These have a time attached to them and this is the first time skew calculation that a peer will have. The hb contains this value and the ref is the uid of the peer that is attached and calculated the skew. The ping and pong values are calculated later when a ping pong sequence of messages are exchange. These are more accurate because there is a larger sample size as the uptime increases. The time is the last time a skew was calculated.

Show which transport links can be used to reach a peer. This table associates a connection fd with a list of peers that are using it. If this connection is lost, then these are peers that may be affected by this event.

Example:

The user is the peer, the path is a list of paths used with the connection fd, and the tport is the transport that contains the connection.

Show the multicast tree for a user or self. This iterates through the adjacency tables by cost and shows the which peers will be reached after each step. The cost increases until all the peers are exhausted. If a U argument is present, then the multicast tree starts from that peer instead of the peer attached to the console.

Example:

The set is an index into the table used for the next hop, this is calculated by transitioning across the transport links. Since the uids are displayed in order, the set may jump back and forth through the table. The alt counter is an alternate path counter. Only the 0 alt path is used, but the others are displayed.

The source is the forwarding peer that sends the message, the tport is the transport local to the source, and dest is the receiver.

Show the transports used to reach a peer for a path. This is the forwarding table that is used to send a message from the local peer to other peers.

Example:

The tport is used for sending a message to dest. The cost is the first hop cost, the path_cost is the total cost through all hops.

Show the forwarding table for a message received from each of the peers. When a message is received from a peer, it may need to be forwarded to other peers to completely cover the network. This shows the forwarding tables for each peer.

Example:

The source is index the forwarding table used, the tport is the transport used to forward the message.

Show what each fd is used for. This iterates the fd tables and shows what each fd is doing.

Example:

The fields are:

Show the buffer usage of each connection. These buffers expand to contain an entire message, since there is no streaming of large messages.

Example:

The fields are:

Show the size and counts of the subject publish and subscribe windows as well as the size of subscription tables and bloom filters.

Example:

The first two are the subscription and publish windows. These tables are rotated to old when they get to win_size with at least interval seconds. The max_size is the largest size of this window.

The inbox entry is a route cache for subjects that have a _INBOX prefix. The route entry is a cache for routes, indexed by subject hash. The bloom entry is the sum of the size of bloom filters for every peer in the network. The rv entry is the subscription table for RV clients attached.

Show where the bloom filters are used for a path. The forwarding table has only one transport entry for each peer, path combination. If a message is forwarded on more than one transport, it is because there are multiple peers that are subscribed across multiple transports for the path. The receiving side also filters the messages through the bloom filters by calculating the ports that are needed for the path to completely cover the network. There may be redundant transports that are inactive for each path either because the cost is more or the path selection prefers one transport over the other.

Example:

Every peer has a bloom filter associated with it. The console, ipc, and sys filters are the local bloom filters which are combined into one filter in another peer. They are split in the local peer so that the traffic destination can be split to the separate processing functions. The sys filter only match the subjects that are used for the system, namely, the _I.<nonce>.> subject and the _M.> subject. The console are the subjects subscribed by the console. The ipc are the subjects subscribed by clients. The all-peers are the combination of all the peers subscriptions, this is used for receiving messages. The individual peer bloom filters are for forwarding messages.

The fields are:

Show which peer bloom filters match a subject. If a message was published with subject S, this shows which peer’s bloom filter would match it. This doesn’t match against the local filters.

Example:

Show the graph description of the network. This creates a description of the network by matching the names of the transports with the names that the peers use. This doesn’t use any network probing, it uses the link state database to calculate the network connectivity. The link state database doesn’t have connection IP addresses associated with it, but it does have a link name and link type. The name/types are enough to describe the network, but doesn’t show how the links are connected to the host with IP addresses.

Example:

The start is the peer attached to the console. The node is the list of peers in the network ordered by age. The following lines have a prefix which is the type of transport used, which is either mesh, tcp, or pgm. The suffix of the type is the name of the transport. Following the "type_name" are the peers which are connected using this transport. If the cost is not the default of 1000, then there will be a : followed by the cost of the transport.

Show the route cache hit and miss statistics. To reduce the number of bloom filters and hash tables that a message must flow through to match the subject, the route for the subject is cached. This cache needs to be updated when a subscription operation occurs, so this purges the entries which are affected by these operations, reducing the cache effectiveness. When a new subject published will also cause a miss. The cache size has a maximum of 256K entries, and when this is hit, the cache is purged and recreated.

Example:

Each tport has a route cache. The hit_pct is a percentage, hit * 100 / total. The hit is how many times an entry was present in the cache, a miss is not present. The max_cnt is the maximum number of cache entries that have occurred since the transport was created. The max_size is the max data size of the entries, which are fds. Some of the entries will have zero size, when there is no route for the subject.

Show the latency of poll states, the average time used for processing timers, read, write, and routing events.

Example:

In a busy router, the read, route, write operations will process multiple messages at a time, depending on how many fit inside of a read buffer. A read buffer is 16KB and is resized only when a large message requires more memory. The sum of these is close to the average latency used by the router per message, even if the time used per message is a fraction of that, since the messages are processed in batches.

The read_cnt is the sum of the counts in the rd_lo and read states, the write_cnt is the sum of the counts in the write, wr_hi, wr_poll states. The difference between rd_lo and read is that the rd_lo state occurs after the read buffer is full or the fd has no more data to read. The wr_hi are the number of times that the write buffer is full. The wr_poll state is the number of times that the fd is part of the poll set because there is back pressure on the connection.

Show the RV host services.

Example:

The svc is the service number, session is the session identifier, user is the user name associated with the session, port is the daemon port number, start is when the host started. The cl is the active number of clients. If the number of clients is zero then the host service is not active, it doesn’t publish any _RV system subjects. The bs, br, ms, mr, idl, odl are the same stats published with the _RV.INFO.SYSTEM.HOST.STATUS.5230FA7C message.

The session ip will be a random address unless configured with the no_fakeip setting, described in Tib RV.

Show the RV subscriptions, which is any subscription that uses an service number. A service name used by another protocol that is not a valid RV service will not have a RV subscriptions.

Example:

The svc field is the service number, the session is a identifier for the connection, which in this case, uses the host prefix and a nanosecond resolution timestamp as the unique identifier. There are other methods used, but they usually have host prefix, a timestamp and/or a process id. The user is derived from the protocol’s method of attaching a user name to the session. The user is often a login name when using RV. The p is set when the subscription is a pattern. The subject is the subscription string.

Show the console rpcs that are currently running. These are created with commands entered into the console or the web interface. These are: "ping", "remote", "show subs", "sub <subject>", "psub <subject>", and "snap <subject>".

Example:

The type is the command, the arg is a subject or a peer name. The recv is the number of messages received, count is the number expected if it is not a subscription type. The cancel command will stop the non-subscription type commands, unsub or punsub commands will stop the subscription type commands.

These commands do pub/sub through the console. The messages have a format attached to them, which is an integer value mapped to decoding methods. If the format is matched with a decoder, then it is decoded to field/value pairs and printed. If a method is not matched, then the value is an opaque string of bytes and that displayed.

Subscribe to subject S. If a file is present, then the publishes are sent to the file instead of printed to the console.

Unsubscribe to subject S. If a file is present, then stop the publishes sent to the file. If only unsub is used, then all subjects are unsubscribed.

Subscribe to RV style wildcard W. If a file is present, then the publishes are sent to the file instead of printed to the console.

Unsubscribe to RV style wildcard W. If a file is present, then stop the publishes sent to the file. If only punsub is used, then all patterns are unsubscribed.

Subscribe to glob style wildcard W. If a file is present, then the publishes are sent to the file instead of printed to the console.

Unsubscribe to glob style wildcard W. If a file is present, then stop the publishes sent to the file.

Publish an empty message to subject S with an _INBOX reply, then wait for the _INBOX subject and print the message received. The _INBOX used is assigned is subscribed by the console automatically.

Send a message M so subscriptions S.

Send a message M to subscriptions S with the trace flag set, which causes any of the intermediate hops as well as the final destination to send an ack reply.

Send a message M to subscription S with the ack flag set, which causes the destinations to send an ack reply.

Send a message M to subscription S with a return inbox.

Randomly choose a subscription match for S and forward message M to that endpoint. This would include both wildcard subscriptions and normal ones.

A cancel command stops any console subscription or RPC, such as ping. This marks the endpoint as canceled, so if results are returned after a cancel, they will be discarded.

This alters the local bloom filter to use a different seed. Changing the bloom filter seed will alter the bits in the hash such that collisions occur at different positions. If a low rate subscription has a collision with a high rate subscription, this would cause unnecessary traffic that can be avoided by altering the bloom filter seed. This doesn’t solve when the 32 bit hashes have collisions, but these are much less likely than a bloom filter collision.

The log messages are normally printed to the console, this mutes them. The log is still present, using the log command will show them and the log file if active, will still be appended. If messages to the console are being printed too fast for the terminal to display them, this will automatically turn on.

This removes the mute for printing log messages to the console.

The integer value is either a mask or a list of strings that turn the debug logging on or off. When debug 0 is used, this turns of the debug messages.

The last 4 don’t have an integer mask because they use different debug variables that the others.

Dumps the current events to a log file for examining later. Useful when a networking problem occurs and is hard to reproduce.

Exit the process without shutting down existing connections and sending bye messages to the network.

Normal shutdown. Existing connections will stop reading new messages send bye messages to connected peers and flush the data in the write queues.