1
0
mirror of https://github.com/mainflux/mainflux.git synced 2025-04-27 13:48:49 +08:00

99 lines
6.2 KiB
Markdown
Raw Normal View History

MF-730 - Add digital twin service for things (#855) * Add starter kit Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add http Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add cmd/main.go Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Remove reference to things from README.md Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add starter kit Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add http Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add cmd/main.go Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Remove reference to things from README.md Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Fix env vars in README.md Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Fix env vars in README.md Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Rename kit to mfxkit Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Rename kit to mfxkit Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add docker compose related files Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add docker compose related files Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add Dockerfile to mfxkit Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add Dockerfile to mfxkit Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add twin service to docker-compose.yml Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add twin service to docker-compose.yml Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add mongo db connection Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add mongo db connection Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add TwinRepository mockup Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Fix docker env vars Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Fix docker env vars Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add twins repo mongodb routines Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add twins repo mongodb routines Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add mongo db docker test suite Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add mongo db docker test suite Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add idp and toDBTwin Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add idp and toDBTwin Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add test cases to repo twin save test Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add test cases to repo twin save test Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add users grpc Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add users grpc Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add methods and tests for udpate and update key Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add methods and tests for udpate and update key Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add methods and tests for remove twin Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add methods and tests for remove twin Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add svc methods to loggin and metrics Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add svc methods to loggin and metrics Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add AddTwin endpoint Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add AddTwin endpoint Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add update endpoints Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add update endpoints Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add view and remove endpoints Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add view and remove endpoints Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add twin repo mock Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add twin repo mock Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add owner arg to twins repo methods Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add owner arg to twins repo methods Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add mock idp service Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add mock users service Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add tests for service Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add tests for service Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add mqtt client to twins service Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add mqtt client to twins service Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add protocol to mqtt string var Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add protocol to mqtt string var Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add time and attributes related data to Twin struct Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add time and attributes related data to Twin struct Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add mqtt publish JSON serialized twin to service Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add mqtt publish JSON serialized twin to service Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add endpoint tests for save and update twin Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add endpoint tests for save and update twin Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add endpoint tests for update key and view thing Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add endpoint tests for update key and view thing Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Fix test error for view twin Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Fix test error for view twin Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add endpoint tests for remove twin Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add endpoint tests for remove twin Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add NATS client Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add NATS client Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add normalizer to nats Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add normalizer to nats Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Refactor nats publish() and remove normalizer Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Refactor nats publish() and remove normalizer Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add ListThingsByChannel() and RetrieveByChannel() Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add ListThingsByChannel() and RetrieveByChannel() Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add Twin struct directly in mongodb Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add Twin struct directly in mongodb Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Change limit type to uint64 Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Change limit type to uint64 Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Compose nats subject from msg ch and subtopic Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Compose nats subject from msg ch and subtopic Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add mqtt notifs for thing creation and key update Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add mqtt notifs for thing creation and key update Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add RetrieveAll to twinRepository and ListTwins to service Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add RetrieveAll to twinRepository and ListTwins to service Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add tests for RetrieveAll and ListTwins Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add tests for RetrieveAll and ListTwins Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Fix Service interface error Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Remove test Ping endpoint Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add mqtt topic to service config Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Remove UpdateKey request and add fields to Twin related requests Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add Attributes and State add and view Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add ListTwins endpoint Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Fix service and database tests Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add ListTwinsByThing endpoint Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Remove twin directory Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add States and Definitions to Twin Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add defer publish to AddTwin Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add refactored and deferred mqtt client publish to Service methods Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add paho subservice Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add paho client to nats Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add mqtt publish and state update Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add Publish wrapper Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Use paho Publish wrapper Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Fix tests Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Save states in separate collection Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Replace []byte payload by []map[string]interface{} Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add offset to PageMetadata Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Rename TwinsPage to Page Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add StateRepository Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add StateRepository, mongodb StateRepository adapter and ListStates endpoint Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add RetrieveLast twin to states repository Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Parse def in nats to update state attrib Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add prepareState() helper Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Replace list things by id by view thing by id Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add state repo mock Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Fix service tests Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Fix mongo twins repo tests and remove owner from RetrieveByID params Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Fix mongo twins repo tests and remove owner from Remove params Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Replace ChannelID in Attribute by Channel Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Fix endpoint tests for add and update twin Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Fix endpoint tests for view and remove twin Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Use new auth service Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Fix mocks auth and mongodb tests Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Fix licensing info Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Fix 'for for' Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Replace short dec by var for default vals and add string map for mqtt op info Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Use Record from mainflux/senml and rename broker to nats in main.go Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Remove Key from Twin Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Remove redundant id uniqueness check from mongodb Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add attrib name to attrib update info Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Fix typos and remove isZeroOfUnderlyingType() helper Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Rename paho to mqtt Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Fix env vars in main.go and README.md Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Move twins/api/twins to twins/api and rename twins- prefix to tw- in main.go Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Remove flows for manual testing and revert to master docker-compose.yml Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Remove publish from nats and fix tests by updating package names Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Rename users.go to authn.go Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Fix crud op names Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Change id to twinID Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Inline if err != nil and change <prefix>ID to <prefix>_id Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Fix metadata db search test Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Rename mqtt and nats related files to publisher and subscriber respectively Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Move save state logic from nats to service Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Fix endpoint tests Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Remove close check from main.go Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Complete deploy section Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add unit tests for states Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Remove debugging artefacts and unneeded comments to exported funcs Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com> * Add port num to defAuthnURL Signed-off-by: Darko Draskovic <darko.draskovic@gmail.com>
2020-01-10 17:25:36 +01:00
# Twins
Service twins is used for CRUD and update of digital twins. Twin is a semantic
representation of a real world entity, be it device, application or something
else. It holds the sequence of attribute based definitions of a real world thing
and refers to the time series of definition based states that hold the
historical data about the represented real world thing.
## Configuration
The service is configured using the environment variables presented in the
following table. Note that any unset variables will be replaced with their
default values.
| Variable | Description | Default |
|----------------------------|----------------------------------------------------------------------|-----------------------|
| MF_TWINS_LOG_LEVEL | Log level for twin service (debug, info, warn, error) | error |
| MF_TWINS_HTTP_PORT | Twins service HTTP port | 9021 |
| MF_TWINS_SERVER_CERT | Path to server certificate in PEM format | |
| MF_TWINS_SERVER_KEY | Path to server key in PEM format | |
| MF_JAEGER_URL | Jaeger server URL | |
| MF_TWINS_DB_NAME | Database name | mainflux |
| MF_TWINS_DB_HOST | Database host address | localhost |
| MF_TWINS_DB_PORT | Database host port | 27017 |
| MF_TWINS_SINGLE_USER_EMAIL | User email for single user mode (no gRPC communication with users) | |
| MF_TWINS_SINGLE_USER_TOKEN | User token for single user mode that should be passed in auth header | |
| MF_TWINS_CLIENT_TLS | Flag that indicates if TLS should be turned on | false |
| MF_TWINS_CA_CERTS | Path to trusted CAs in PEM format | |
| MF_TWINS_MQTT_URL | Mqtt broker URL for twin CRUD and states update notifications | tcp://localhost:1883 |
| MF_TWINS_THING_ID | ID of thing representing twins service & mqtt user | |
| MF_TWINS_THING_KEY | Key of thing representing twins service & mqtt pass | |
| MF_TWINS_CHANNEL_ID | Mqtt notifications topic | |
| MF_NATS_URL | Mainflux NATS broker URL | nats://127.0.0.1:4222 |
| MF_AUTHN_GRPC_PORT | Authn service gRPC port | 8181 |
| MF_AUTHN_TIMEOUT | Authn gRPC request timeout in seconds | 1 |
| MF_AUTHN_URL | Authn service URL | localhost:8181 |
## Deployment
The service itself is distributed as Docker container. The following snippet
provides a compose file template that can be used to deploy the service container
locally:
```yaml
version: "3"
services:
twins:
image: mainflux/twins:[version]
container_name: [instance name]
ports:
- [host machine port]:[configured HTTP port]
environment:
MF_TWINS_LOG_LEVEL: [Twins log level]
MF_TWINS_HTTP_PORT: [Service HTTP port]
MF_TWINS_SERVER_CERT: [String path to server cert in pem format]
MF_TWINS_SERVER_KEY: [String path to server key in pem format]
MF_JAEGER_URL: [Jaeger server URL]
MF_TWINS_DB_NAME: [Database name]
MF_TWINS_DB_HOST: [Database host address]
MF_TWINS_DB_PORT: [Database host port]
MF_TWINS_SINGLE_USER_EMAIL: [User email for single user mode]
MF_TWINS_SINGLE_USER_TOKEN: [User token for single user mode]
MF_TWINS_CLIENT_TLS: [Flag that indicates if TLS should be turned on]
MF_TWINS_CA_CERTS: [Path to trusted CAs in PEM format]
MF_TWINS_MQTT_URL: [Mqtt broker URL for twin CRUD and states]
MF_TWINS_THING_ID: [ID of thing representing twins service]
MF_TWINS_THING_KEY: [Key of thing representing twins service]
MF_TWINS_CHANNEL_ID: [Mqtt notifications topic]
MF_NATS_URL: [Mainflux NATS broker URL]
MF_AUTHN_GRPC_PORT: [Authn service gRPC port]
MF_AUTHN_TIMEOUT: [Authn gRPC request timeout in seconds]
MF_AUTHN_URL: [Authn service URL]
```
To start the service outside of the container, execute the following shell script:
```bash
# download the latest version of the service
go get github.com/mainflux/mainflux
cd $GOPATH/src/github.com/mainflux/mainflux
# compile the twins
make twins
# copy binary to bin
make install
# set the environment variables and run the service
MF_TWINS_LOG_LEVEL: [Twins log level] MF_TWINS_HTTP_PORT: [Service HTTP port] MF_TWINS_SERVER_CERT: [String path to server cert in pem format] MF_TWINS_SERVER_KEY: [String path to server key in pem format] MF_JAEGER_URL: [Jaeger server URL] MF_TWINS_DB_NAME: [Database name] MF_TWINS_DB_HOST: [Database host address] MF_TWINS_DB_PORT: [Database host port] MF_TWINS_SINGLE_USER_EMAIL: [User email for single user mode] MF_TWINS_SINGLE_USER_TOKEN: [User token for single user mode] MF_TWINS_CLIENT_TLS: [Flag that indicates if TLS should be turned on] MF_TWINS_CA_CERTS: [Path to trusted CAs in PEM format] MF_TWINS_MQTT_URL: [Mqtt broker URL for twin CRUD and states] MF_TWINS_THING_ID: [ID of thing representing twins service] MF_TWINS_THING_KEY: [Key of thing representing twins service] MF_TWINS_CHANNEL_ID: [Mqtt notifications topic] MF_NATS_URL: [Mainflux NATS broker URL] MF_AUTHN_GRPC_PORT: [Authn service gRPC port] MF_AUTHN_TIMEOUT: [Authn gRPC request timeout in seconds] MF_AUTHN_URL: [Authn service URL] $GOBIN/mainflux-twins
```
## Usage
For more information about service capabilities and its usage, please check out
the [API documentation](swagger.yaml).
[doc]: http://mainflux.readthedocs.io