1
0
mirror of https://github.com/mainflux/mainflux.git synced 2025-04-26 13:48:53 +08:00

96 lines
4.2 KiB
Markdown
Raw Normal View History

# Postgres writer
Postgres writer provides message repository implementation for Postgres.
## Configuration
The service is configured using the environment variables presented in the
following table. Note that any unset variables will be replaced with their
default values.
| Variable | Description | Default |
|-------------------------------------|--------------------------------------------|------------------------|
| MF_NATS_URL | NATS instance URL | nats://localhost:4222 |
| MF_POSTGRES_WRITER_LOG_LEVEL | Service log level | error |
| MF_POSTGRES_WRITER_PORT | Service HTTP port | 9104 |
| MF_POSTGRES_WRITER_DB_HOST | Postgres DB host | postgres |
| MF_POSTGRES_WRITER_DB_PORT | Postgres DB port | 5432 |
| MF_POSTGRES_WRITER_DB_USER | Postgres user | mainflux |
| MF_POSTGRES_WRITER_DB_PASS | Postgres password | mainflux |
| MF_POSTGRES_WRITER_DB | Postgres database name | messages |
| MF_POSTGRES_WRITER_DB_SSL_MODE | Postgres SSL mode | disabled |
| MF_POSTGRES_WRITER_DB_SSL_CERT | Postgres SSL certificate path | "" |
| MF_POSTGRES_WRITER_DB_SSL_KEY | Postgres SSL key | "" |
| MF_POSTGRES_WRITER_DB_SSL_ROOT_CERT | Postgres SSL root certificate path | "" |
| MF_POSTGRES_WRITER_SUBJECTS_CONFIG | Configuration file path with subjects list | /config/subjects.toml |
| MF_POSTGRES_WRITER_CONTENT_TYPE | Message payload Content Type | application/senml+json |
## Deployment
```yaml
MF-780 - Use Normalizer as a lib (#915) * Use Normalizer as a lib To normalize messages on the consumer side, Normalizer is moved to the internal pkgs. Writers being message consumers are modified to do message normalization instead of subscribing to normalized messages subject. Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com> * Fix logging middleware for readers and writers Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com> * Remove normalizer interface Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com> * Use Normalizer in writers As we agreed on #919, we'll use normalizer as an interface and provide the default SenML implementation. Because of that, Normalizer is removed from `internal` and we'll use the project structure proposed in the aforementioned issue. Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com> * Fix tests Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com> * Remove unused batch settings from influxDB reader Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com> * Update docs Move Normalizer service to `addons`. Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com> * Rename channels input topic Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com> * Update Noramlizer docs Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com> * Remove commented code Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com> * Update readers logging Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com> * Update addons docker-compose files Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com> * Update topcis explanations Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>
2019-10-31 14:04:47 +01:00
version: "3.7"
postgres-writer:
image: mainflux/postgres-writer:[version]
container_name: [instance name]
depends_on:
- postgres
- nats
restart: on-failure
environment:
MF_NATS_URL: [NATS instance URL]
MF_POSTGRES_WRITER_LOG_LEVEL: [Service log level]
MF_POSTGRES_WRITER_PORT: [Service HTTP port]
MF_POSTGRES_WRITER_DB_HOST: [Postgres host]
MF_POSTGRES_WRITER_DB_PORT: [Postgres port]
MF_POSTGRES_WRITER_DB_USER: [Postgres user]
MF_POSTGRES_WRITER_DB_PASS: [Postgres password]
MF-1096 - Fix AuthN and Things Auth ENVARS (#1066) * NOISSUE - Fix AuthN and Things Auth ENVARS Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com> * Add DB envars to env.go Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com> * Fix envars Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com> * Add DefLogLLevelError Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com> * Fix DB names Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com> * Fix DB names Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com> * Fix DB names and HTTP ports Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com> * Fix .env Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com> * Revert writers DB names Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com> * Rm unused Twins envars Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com> * Fix reviews Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com> * Fix reviews Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com> * Fix typo Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com> * Remove definitions from env.go Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com> * Revert HTTP gRPC ports Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com> * Use default NATS URL as string Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com> * Fix reviews Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com> * Revert default ports Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com> * Fix typo Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com> * Fix cassandra ENVARS Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com> * Fix cassandra reader ENVARS Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com> * Fix readers and writers envars Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com> * Fix readers and writers .env Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com> * Fix reviews Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>
2020-04-07 12:02:17 +02:00
MF_POSTGRES_WRITER_DB: [Postgres database name]
MF_POSTGRES_WRITER_DB_SSL_MODE: [Postgres SSL mode]
MF_POSTGRES_WRITER_DB_SSL_CERT: [Postgres SSL cert]
MF_POSTGRES_WRITER_DB_SSL_KEY: [Postgres SSL key]
MF_POSTGRES_WRITER_DB_SSL_ROOT_CERT: [Postgres SSL Root cert]
NOISSUE - Adding subtopics filtering in writer services (#1072) * Add feature of filtering by subtopics in writer Signed-off-by: Jonathan Dreyer <jonathan.dreyer@cleanenergie.ch> * Fix mistake Signed-off-by: Jonathan Dreyer <jonathan.dreyer@cleanenergie.ch> * Refactoring writer sevices Signed-off-by: Jonathan Dreyer <jonathan.dreyer@cleanenergie.ch> * Rename variables related to filter (channels & subtopics) Signed-off-by: Jonathan Dreyer <jonathan.dreyer@cleanenergie.ch> * Set default value of filtering when configuration file doesn't exist Signed-off-by: Jonathan Dreyer <jonathan.dreyer@cleanenergie.ch> * Add a blank line at the end of the file Signed-off-by: Jonathan Dreyer <jonathan.dreyer@cleanenergie.ch> * Refactor loading filters configuration (moving into writer package, merge both loading methods & returning error) Signed-off-by: Jonathan Dreyer <jonathan.dreyer@cleanenergie.ch> * Remove useless log Signed-off-by: Jonathan Dreyer <jonathan.dreyer@cleanenergie.ch> * Change type of variables (channels & subtopics) and simplify loading method Signed-off-by: Jonathan Dreyer <jonathan.dreyer@cleanenergie.ch> * Add logging error when loading filters Signed-off-by: Jonathan Dreyer <jonathan.dreyer@cleanenergie.ch> * Simplify return configuration in loading method Signed-off-by: Jonathan Dreyer <jonathan.dreyer@cleanenergie.ch> * Merge both filter files into one file Signed-off-by: Jonathan Dreyer <jonathan.dreyer@cleanenergie.ch> * Move loading subjects into writer package Signed-off-by: Jonathan Dreyer <jonathan.dreyer@cleanenergie.ch> * Add subscribe to selected subjects Signed-off-by: Jonathan Dreyer <jonathan.dreyer@cleanenergie.ch> * Edit README of writer services Signed-off-by: Jonathan Dreyer <jonathan.dreyer@cleanenergie.ch> * Keep only subscribe loop Signed-off-by: Jonathan Dreyer <jonathan.dreyer@cleanenergie.ch> * Use full NATS subjects Signed-off-by: Jonathan Dreyer <jonathan.dreyer@cleanenergie.ch> * Edit comment in subjects files Signed-off-by: Jonathan Dreyer <jonathan.dreyer@cleanenergie.ch> Co-authored-by: Drasko DRASKOVIC <drasko.draskovic@gmail.com>
2020-03-30 18:24:27 +02:00
MF_POSTGRES_WRITER_SUBJECTS_CONFIG: [Configuration file path with subjects list]
MF_POSTGRES_WRITER_CONTENT_TYPE: [Message payload Content Type]
ports:
- 9104:9104
networks:
- docker_mainflux-base-net
volume:
NOISSUE - Adding subtopics filtering in writer services (#1072) * Add feature of filtering by subtopics in writer Signed-off-by: Jonathan Dreyer <jonathan.dreyer@cleanenergie.ch> * Fix mistake Signed-off-by: Jonathan Dreyer <jonathan.dreyer@cleanenergie.ch> * Refactoring writer sevices Signed-off-by: Jonathan Dreyer <jonathan.dreyer@cleanenergie.ch> * Rename variables related to filter (channels & subtopics) Signed-off-by: Jonathan Dreyer <jonathan.dreyer@cleanenergie.ch> * Set default value of filtering when configuration file doesn't exist Signed-off-by: Jonathan Dreyer <jonathan.dreyer@cleanenergie.ch> * Add a blank line at the end of the file Signed-off-by: Jonathan Dreyer <jonathan.dreyer@cleanenergie.ch> * Refactor loading filters configuration (moving into writer package, merge both loading methods & returning error) Signed-off-by: Jonathan Dreyer <jonathan.dreyer@cleanenergie.ch> * Remove useless log Signed-off-by: Jonathan Dreyer <jonathan.dreyer@cleanenergie.ch> * Change type of variables (channels & subtopics) and simplify loading method Signed-off-by: Jonathan Dreyer <jonathan.dreyer@cleanenergie.ch> * Add logging error when loading filters Signed-off-by: Jonathan Dreyer <jonathan.dreyer@cleanenergie.ch> * Simplify return configuration in loading method Signed-off-by: Jonathan Dreyer <jonathan.dreyer@cleanenergie.ch> * Merge both filter files into one file Signed-off-by: Jonathan Dreyer <jonathan.dreyer@cleanenergie.ch> * Move loading subjects into writer package Signed-off-by: Jonathan Dreyer <jonathan.dreyer@cleanenergie.ch> * Add subscribe to selected subjects Signed-off-by: Jonathan Dreyer <jonathan.dreyer@cleanenergie.ch> * Edit README of writer services Signed-off-by: Jonathan Dreyer <jonathan.dreyer@cleanenergie.ch> * Keep only subscribe loop Signed-off-by: Jonathan Dreyer <jonathan.dreyer@cleanenergie.ch> * Use full NATS subjects Signed-off-by: Jonathan Dreyer <jonathan.dreyer@cleanenergie.ch> * Edit comment in subjects files Signed-off-by: Jonathan Dreyer <jonathan.dreyer@cleanenergie.ch> Co-authored-by: Drasko DRASKOVIC <drasko.draskovic@gmail.com>
2020-03-30 18:24:27 +02:00
- ./subjects.yaml:/config/subjects.yaml
```
To start the service, execute the following shell script:
```bash
# download the latest version of the service
git clone https://github.com/mainflux/mainflux
cd mainflux
# compile the postgres writer
make postgres-writer
# copy binary to bin
make install
# Set the environment variables and run the service
MF-1096 - Fix AuthN and Things Auth ENVARS (#1066) * NOISSUE - Fix AuthN and Things Auth ENVARS Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com> * Add DB envars to env.go Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com> * Fix envars Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com> * Add DefLogLLevelError Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com> * Fix DB names Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com> * Fix DB names Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com> * Fix DB names and HTTP ports Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com> * Fix .env Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com> * Revert writers DB names Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com> * Rm unused Twins envars Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com> * Fix reviews Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com> * Fix reviews Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com> * Fix typo Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com> * Remove definitions from env.go Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com> * Revert HTTP gRPC ports Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com> * Use default NATS URL as string Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com> * Fix reviews Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com> * Revert default ports Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com> * Fix typo Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com> * Fix cassandra ENVARS Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com> * Fix cassandra reader ENVARS Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com> * Fix readers and writers envars Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com> * Fix readers and writers .env Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com> * Fix reviews Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>
2020-04-07 12:02:17 +02:00
MF_NATS_URL=[NATS instance URL] \
MF_POSTGRES_WRITER_LOG_LEVEL=[Service log level] \
MF_POSTGRES_WRITER_PORT=[Service HTTP port] \
MF_POSTGRES_WRITER_DB_HOST=[Postgres host] \
MF_POSTGRES_WRITER_DB_PORT=[Postgres port] \
MF_POSTGRES_WRITER_DB_USER=[Postgres user] \
MF_POSTGRES_WRITER_DB_PASS=[Postgres password] \
MF_POSTGRES_WRITER_DB=[Postgres database name] \
MF_POSTGRES_WRITER_DB_SSL_MODE=[Postgres SSL mode] \
MF_POSTGRES_WRITER_DB_SSL_CERT=[Postgres SSL cert] \
MF_POSTGRES_WRITER_DB_SSL_KEY=[Postgres SSL key] \
MF_POSTGRES_WRITER_DB_SSL_ROOT_CERT=[Postgres SSL Root cert] \
MF_POSTGRES_WRITER_SUBJECTS_CONFIG=[Configuration file path with subjects list] \
$GOBIN/mainflux-postgres-writer
```
## Usage
Starting service will start consuming normalized messages in SenML format.