OrgGo/Mainflux.mainflux
1
0
Fork 0
mirror of https://github.com/mainflux/mainflux.git synced 2025-04-28 13:48:49 +08:00
Code Issues Projects Releases Wiki Activity

MF-698 - Add missing info and docs about sys event sourcing (#712)

Browse Source 
* Add event sourcing section in dev guide

Signed-off-by: Aleksandar Novakovic <aleksandar.novakovic@mainflux.com>

* Fix spelling in the docs

Signed-off-by: Aleksandar Novakovic <aleksandar.novakovic@mainflux.com>

* Fix notice formatting in docs

Signed-off-by: Aleksandar Novakovic <aleksandar.novakovic@mainflux.com>

* Fix typos in docs

Signed-off-by: Aleksandar Novakovic <aleksandar.novakovic@mainflux.com>

* Add highlight for event type

Signed-off-by: Aleksandar Novakovic <aleksandar.novakovic@mainflux.com>
This commit is contained in:
Aleksandar Novaković 2019-04-16 17:28:43 +02:00 committed by Nikola Marčetić
parent 5799356b14
commit de736a5c47
1 changed files with 308 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

308
docs/dev-guide.md
View File

@ -255,3 +255,311 @@ which will properly configure and run all microservices.
Please assure that MQTT microservice has `node_modules` installed, as explained in _MQTT Microservice_ chapter.
> N.B. `make rundev` actually calls helper script `scripts/run.sh`, so you can inspect this script for the details.
## Events
In order to be easily integratable system, Mainflux is using [Redis Streams](https://redis.io/topics/streams-intro)
as an event log for event sourcing. Services that are publishing events to Redis Streams
are `things` service, `bootstrap` service, and `mqtt` adapter.
### Things Service
For every operation that has side effects (that is changing service state) `things`
service will generate new event and publish it to Redis Stream called `mainflux.things`.
Every event has its own event ID that is automatically generated and `operation`
field that can have one of the following values:
- `thing.create` for thing creation,
- `thing.update` for thing update,
- `thing.remove` for thing removal,
- `thing.connect` for connecting a thing to a channel,
- `thing.disconnect` for disconnecting thing from a channel,
- `channel.create` for channel creation,
- `channel.update` for channel update,
- `channel.remove` for channel removal.
By fetching and processing these events you can reconstruct `things` service state.
If you store some of your custom data in `metadata` field, this is the perfect
way to fetch it and process it. If you want to integrate through
[docker-compose.yml](https://github.com/mainflux/mainflux/blob/master/docker/docker-compose.yml)
you can use `mainflux-things-redis` service. Just connect to it and consume events
from Redis Stream named `mainflux.things`.
#### Thing create event
Whenever thing is created, `things` service will generate new `create` event. This
event will have the following format:
```
1) "1555334740911-0"
2) 1) "type"
2) "device"
3) "operation"
4) "thing.create"
5) "name"
6) "d0"
7) "id"
8) "3c36273a-94ea-4802-84d6-a51de140112e"
9) "owner"
10) "john.doe@email.com"
11) "metadata"
12) "{}"
```
As you can see from this example, every odd field represents field name while every
even field represents field value. This is standard event format for Redis Streams.
If you want to extract `metadata` field from this event, you'll have to read it as
string first, and then you can deserialize it to some structured format.
#### Thing update event
Whenever thing instance is updated, `things` service will generate new `update` event.
This event will have the following format:
```
1) "1555336161544-0"
2) 1) "operation"
2) "thing.update"
3) "name"
4) "weio"
5) "id"
6) "3c36273a-94ea-4802-84d6-a51de140112e"
7) "type"
8) "device"
```
Note that thing update event will contain only those fields that were updated using
update endpoint.
#### Thing remove event
Whenever thing instance is removed from the system, `things` service will generate and
publish new `remove` event. This event will have the following format:
```
1) 1) "1555339313003-0"
2) 1) "id"
2) "3c36273a-94ea-4802-84d6-a51de140112e"
3) "operation"
4) "thing.remove"
```
#### Channel create event
Whenever channel instance is created, `things` service will generate and publish new
`create` event. This event will have the following format:
```
1) "1555334740918-0"
2) 1) "id"
2) "16fb2748-8d3b-4783-b272-bb5f4ad4d661"
3) "owner"
4) "john.doe@email.com"
5) "operation"
6) "channel.create"
7) "name"
8) "c1"
```
#### Channel update event
Whenever channel instance is updated, `things` service will generate and publish new
`update` event. This event will have the following format:
```
1) "1555338870341-0"
2) 1) "name"
2) "chan"
3) "id"
4) "d9d8f31b-f8d4-49c5-b943-6db10d8e2949"
5) "operation"
6) "channel.update"
```
Note that update channel event will contain only those fields that were updated using
update channel endpoint.
#### Channel remove event
Whenever channel instance is removed from the system, `things` service will generate and
publish new `remove` event. This event will have the following format:
```
1) 1) "1555339429661-0"
2) 1) "id"
2) "d9d8f31b-f8d4-49c5-b943-6db10d8e2949"
3) "operation"
4) "channel.remove"
```
#### Connect thing to a channel event
Whenever thing is connected to a channel on `things` service, `things` service will
generate and publish new `connect` event. This event will have the following format:
```
1) "1555334740920-0"
2) 1) "chan_id"
2) "d9d8f31b-f8d4-49c5-b943-6db10d8e2949"
3) "thing_id"
4) "3c36273a-94ea-4802-84d6-a51de140112e"
5) "operation"
6) "thing.connect"
```
#### Disconnect thing from a channel event
Whenever thing is disconnected from a channel on `things` service, `things` service
will generate and publish new `disconnect` event. This event will have the following
format:
```
1) "1555334740920-0"
2) 1) "chan_id"
2) "d9d8f31b-f8d4-49c5-b943-6db10d8e2949"
3) "thing_id"
4) "3c36273a-94ea-4802-84d6-a51de140112e"
5) "operation"
6) "thing.disconnect"
```
> **Note:** Every one of these events will omit fields that were not used or are not
relevant for specific operation. Also, field ordering is not guaranteed, so DO NOT
rely on it.
### Bootstrap Service
Bootstrap service publishes events to Redis Stream called `mainflux.bootstrap`.
Every event from this service contains `operation` field which indicates one of
the following event types:
- `config.create` for configuration creation,
- `config.update` for configuration update,
- `config.remove` for configuration removal,
- `thing.bootstrap` for device bootstrap,
- `thing.state_change` for device state change,
- `thing.update_connections` for device connection update.
If you want to integrate through
[docker-compose.yml](https://github.com/mainflux/mainflux/blob/master/docker/addons/bootstrap/docker-compose.yml)
you can use `mainflux-bootstrap-redis` service. Just connect to it and consume events
from Redis Stream named `mainflux.bootstrap`.
#### Configuration create event
Whenever configuration is created, `bootstrap` service will generate and publish
new `create` event. This event will have the following format:
```
1) "1555404899581-0"
2) 1) "owner"
2) "john.doe@email.com"
3) "name"
4) "some"
5) "channels"
6) "ff13ca9c-7322-4c28-a25c-4fe5c7b753fc, c3642289-501d-4974-82f2-ecccc71b2d82, c3642289-501d-4974-82f2-ecccc71b2d83, cd4ce940-9173-43e3-86f7-f788e055eb14"
7) "externalID"
8) "9c:b6:d:eb:9f:fd"
9) "content"
10) "{}"
11) "timestamp"
12) "1555404899"
13) "operation"
14) "config.create"
15) "id"
16) "63a110d4-2b77-48d2-aa46-2582681eeb82"
```
#### Configuration update event
Whenever configuration is updated, `bootstrap` service will generate and publish
new `update` event. This event will have the following format:
```
1) "1555405104368-0"
2) 1) "content"
2) "NOV_MGT_HOST: http://127.0.0.1:7000\nDOCKER_MGT_HOST: http://127.0.0.1:2375\nAGENT_MGT_HOST: https://127.0.0.1:7003\nMF_MQTT_HOST: tcp://104.248.142.133:8443"
3) "timestamp"
4) "1555405104"
5) "operation"
6) "config.update"
7) "id"
8) "63a110d4-2b77-48d2-aa46-2582681eeb82"
9) "name"
10) "weio"
```
#### Configuration remove event
Whenever configuration is removed, `bootstrap` service will generate and publish
new `remove` event. This event will have the following format:
```
1) "1555405464328-0"
2) 1) "id"
2) "63a110d4-2b77-48d2-aa46-2582681eeb82"
3) "timestamp"
4) "1555405464"
5) "operation"
6) "config.remove"
```
#### Thing bootstrap event
Whenever thing is bootstrapped, `bootstrap` service will generate and publish
new `bootstrap` event. This event will have the following format:
```
1) "1555405173785-0"
2) 1) "externalID"
2) "9c:b6:d:eb:9f:fd"
3) "successful"
4) "1"
5) "timestamp"
6) "1555405173"
7) "operation"
8) "thing.bootstrap"
```
#### Thing change state event
Whenever thing's state changes, `bootstrap` service will generate and publish
new `change state` event. This event will have the following format:
```
1) "1555405294806-0"
2) 1) "id"
2) "63a110d4-2b77-48d2-aa46-2582681eeb82"
3) "state"
4) "0"
5) "timestamp"
6) "1555405294"
7) "operation"
8) "thing.state_change"
```
#### Thing update connections event
Whenever thing's list of connections is updated, `bootstrap` service will generate
and publish new `update connections` event. This event will have the following format:
```
1) "1555405373360-0"
2) 1) "operation"
2) "thing.update_connections"
3) "id"
4) "63a110d4-2b77-48d2-aa46-2582681eeb82"
5) "channels"
6) "ff13ca9c-7322-4c28-a25c-4fe5c7b753fc, 925461e6-edfb-4755-9242-8a57199b90a5, c3642289-501d-4974-82f2-ecccc71b2d82"
7) "timestamp"
8) "1555405373"
```
### MQTT Adapter
Instead of using heartbeat to know when client is connected through MQTT adapter one
can fetch events from Redis Streams that MQTT adapter publishes. MQTT adapter
publishes events every time client connects and disconnects to stream named `mainflux.mqtt`.
Events that are coming from MQTT adapter have following fields:
- `thing_id` ID of a thing that has connected to MQTT adapter,
- `timestamp` is in Epoch UNIX Time Stamp format,
- `event_type` can have two possible values, connect and disconnect,
- `instance` represents MQTT adapter instance.
If you want to integrate through
[docker-compose.yml](https://github.com/mainflux/mainflux/blob/master/docker/docker-compose.yml)
you can use `mainflux-mqtt-redis` service. Just connect to it and consume events
from Redis Stream named `mainflux.mqtt`.
Example of connect event:
```
1) 1) "1555351214144-0"
2) 1) "thing_id"
2) "1c597a85-b68e-42ff-8ed8-a3a761884bc4"
3) "timestamp"
4) "1555351214"
5) "event_type"
6) "connect"
7) "instance"
8) "mqtt-adapter-1"
```
Example of disconnect event:
```
1) 1) "1555351214188-0"
2) 1) "thing_id"
2) "1c597a85-b68e-42ff-8ed8-a3a761884bc4"
3) "timestamp"
4) "1555351214"
5) "event_type"
6) "disconnect"
7) "instance"
8) "mqtt-adapter-1"
```