mirror of
https://github.com/mainflux/mainflux.git
synced 2025-05-02 22:17:10 +08:00

* Upgrade InfluxDB from 1.x to 2.x Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Influx DB configuration updated Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Connection to InfluxDBv2 Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Token cannot be created Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Connected to InfluxDB2 Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Connected to InfluxDB2 Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * InfluxDB v2 Consumer Implementation Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * quickfix Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Influxdb-Writer Unit Tests Update Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Consumer Update Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * InfluxDB Writer Tests Implemented Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * fix Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * InfluxDB Connection Check Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Resolving Remarks Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Resolved consumer-test remark Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * consumer-test slow working version Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * reader changes Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Consumer tests time issue fixed Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Eof warning fixed Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Resolved Reviews Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Makefile Fixed Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Resolved Reviews Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Resolved Reviews Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Resolved Reviews Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Resolved Reviews Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Resolved Reviews Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Readers Initial Setup Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * consumer json fix Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * readers simple version Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * readers simple version fix Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * readers simple version fix Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * readers simple version fix Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * readers simple version fix Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * readers simple version fix Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * readers simple version fix Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Influxdb Reader Parsers Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Influxdb Reader Parsers Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Influxdb Reader Parsers Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Bugfix and resolves comments. Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * one test fails Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * solved last page read Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * writers future time problem fixed Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * weird Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * weird Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Tests Passes Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Resolve Semaphore Issues Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * UUID comment on consumer tests resolved Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Inclusive from and Exclusive to Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Inclusive from and Exclusive to Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * 1 second limits Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * 1 second limits Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * resolved review Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * resolved review Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * resolved review Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * resolved review Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * fixed json time Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * fixed CI error Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * resolved request Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * resolved request Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * resolved requests Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * removed blank line Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * resolved comment Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * deleted unnecessary string builder Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * initial commit Signed-off-by: aryan <aryangodara03@gmail.com> * change influxdb docker image version. Signed-off-by: aryan <aryangodara03@gmail.com> * go mod and vendor fixing Signed-off-by: aryan <aryangodara03@gmail.com> * Upgrade InfluxDB from 1.x to 2.x Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Influx DB configuration updated Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Connection to InfluxDBv2 Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Token cannot be created Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Connected to InfluxDB2 Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Connected to InfluxDB2 Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * InfluxDB v2 Consumer Implementation Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * quickfix Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Influxdb-Writer Unit Tests Update Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Consumer Update Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * InfluxDB Writer Tests Implemented Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * fix Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * InfluxDB Connection Check Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Resolving Remarks Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Resolved consumer-test remark Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * consumer-test slow working version Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * reader changes Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Consumer tests time issue fixed Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Resolved Reviews Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Resolved Reviews Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Resolved Reviews Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Resolved Reviews Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Resolved Reviews Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Resolved Reviews Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Readers Initial Setup Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * consumer json fix Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * readers simple version Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * readers simple version fix Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * readers simple version fix Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * readers simple version fix Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * readers simple version fix Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * readers simple version fix Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * readers simple version fix Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Influxdb Reader Parsers Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Influxdb Reader Parsers Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Influxdb Reader Parsers Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Bugfix and resolves comments. Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * one test fails Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * solved last page read Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * writers future time problem fixed Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * weird Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * weird Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Tests Passes Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Resolve Semaphore Issues Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * UUID comment on consumer tests resolved Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Inclusive from and Exclusive to Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Inclusive from and Exclusive to Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * 1 second limits Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * 1 second limits Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * resolved review Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * resolved review Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * resolved review Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * resolved review Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * fixed json time Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * fixed CI error Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * resolved request Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * resolved request Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * resolved requests Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * removed blank line Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * resolved comment Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * deleted unnecessary string builder Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * initial commit Signed-off-by: aryan <aryangodara03@gmail.com> * change influxdb docker image version. Signed-off-by: aryan <aryangodara03@gmail.com> * go mod and vendor fixing Signed-off-by: aryan <aryangodara03@gmail.com> * go mod fixing Signed-off-by: aryan <aryangodara03@gmail.com> * make tests pass locally, fix errors Signed-off-by: aryan <aryangodara03@gmail.com> * rem unsused variables/consts Signed-off-by: aryan <aryangodara03@gmail.com> * go mod tidy vendor Signed-off-by: aryan <aryangodara03@gmail.com> * add env var and data types Signed-off-by: aryan <aryangodara03@gmail.com> * update influxdb version to latest (2.3 to 2.12) Signed-off-by: aryan <aryangodara03@gmail.com> * change time precision for message tests Signed-off-by: aryan <aryangodara03@gmail.com> * renamed influxdb2 to influxdata Signed-off-by: aryan <aryangodara03@gmail.com> * address remarks on PR Signed-off-by: aryan <aryangodara03@gmail.com> * update influxbd according to latest mf commit Signed-off-by: aryan <aryangodara03@gmail.com> * temp commit, rem before pushing Signed-off-by: aryan <aryangodara03@gmail.com> * update main files and cassandra-reader messages_tests Signed-off-by: aryan <aryangodara03@gmail.com> * fix name of logger while importing Signed-off-by: aryan <aryangodara03@gmail.com> * remove unnecessary print lines Signed-off-by: aryan <aryangodara03@gmail.com> * correct env var name Signed-off-by: aryan <aryangodara03@gmail.com> * change to async consume Signed-off-by: aryan <aryangodara03@gmail.com> * add option to switch bw sync and async Signed-off-by: aryan <aryangodara03@gmail.com> * test for both async and sync Signed-off-by: aryan <aryangodara03@gmail.com> * update consumer and add writeAPIs to config Signed-off-by: aryan <aryangodara03@gmail.com> * revert back to sync consuming Signed-off-by: aryan <aryangodara03@gmail.com> * temp fix for default timeout value Signed-off-by: aryan <aryangodara03@gmail.com> * set default timeout in config. Signed-off-by: aryan <aryangodara03@gmail.com> * remove unwanted env vars, add required ones. Signed-off-by: aryan <aryangodara03@gmail.com> * rem unused username password from config Signed-off-by: aryan <aryangodara03@gmail.com> * update readme, env vars, and remove grafana Signed-off-by: aryan <aryangodara03@gmail.com> * update readme Signed-off-by: aryan <aryangodara03@gmail.com> * fix typo Signed-off-by: aryan <aryangodara03@gmail.com> * update readme description. Signed-off-by: aryan <aryangodara03@gmail.com> * fix more typos. Signed-off-by: aryan <aryangodara03@gmail.com> * add link to official docs to readme. Signed-off-by: aryan <aryangodara03@gmail.com> --------- Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> Signed-off-by: aryan <aryangodara03@gmail.com> Co-authored-by: fatih <fatihdurmaz@sabanciuniv.edu> Co-authored-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>
200 lines
6.9 KiB
Go
200 lines
6.9 KiB
Go
// Copyright 2020-2021 InfluxData, Inc. All rights reserved.
|
|
// Use of this source code is governed by MIT
|
|
// license that can be found in the LICENSE file.
|
|
|
|
package write
|
|
|
|
import (
|
|
"time"
|
|
)
|
|
|
|
// Options holds write configuration properties
|
|
type Options struct {
|
|
// Maximum number of points sent to server in single request. Default 5000
|
|
batchSize uint
|
|
// Interval, in ms, in which is buffer flushed if it has not been already written (by reaching batch size) . Default 1000ms
|
|
flushInterval uint
|
|
// Precision to use in writes for timestamp. In unit of duration: time.Nanosecond, time.Microsecond, time.Millisecond, time.Second
|
|
// Default time.Nanosecond
|
|
precision time.Duration
|
|
// Whether to use GZip compression in requests. Default false
|
|
useGZip bool
|
|
// Tags added to each point during writing. If a point already has a tag with the same key, it is left unchanged.
|
|
defaultTags map[string]string
|
|
// Default retry interval in ms, if not sent by server. Default 5,000.
|
|
retryInterval uint
|
|
// Maximum count of retry attempts of failed writes, default 5.
|
|
maxRetries uint
|
|
// Maximum number of points to keep for retry. Should be multiple of BatchSize. Default 50,000.
|
|
retryBufferLimit uint
|
|
// The maximum delay between each retry attempt in milliseconds, default 125,000.
|
|
maxRetryInterval uint
|
|
// The maximum total retry timeout in millisecond, default 180,000.
|
|
maxRetryTime uint
|
|
// The base for the exponential retry delay
|
|
exponentialBase uint
|
|
// InfluxDB Enterprise write consistency as explained in https://docs.influxdata.com/enterprise_influxdb/v1.9/concepts/clustering/#write-consistency
|
|
consistency Consistency
|
|
}
|
|
|
|
const (
|
|
// ConsistencyOne requires at least one data node acknowledged a write.
|
|
ConsistencyOne Consistency = "one"
|
|
|
|
// ConsistencyAll requires all data nodes to acknowledge a write.
|
|
ConsistencyAll Consistency = "all"
|
|
|
|
// ConsistencyQuorum requires a quorum of data nodes to acknowledge a write.
|
|
ConsistencyQuorum Consistency = "quorum"
|
|
|
|
// ConsistencyAny allows for hinted hand off, potentially no write happened yet.
|
|
ConsistencyAny Consistency = "any"
|
|
)
|
|
|
|
// Consistency defines enum for allows consistency values for InfluxDB Enterprise, as explained https://docs.influxdata.com/enterprise_influxdb/v1.9/concepts/clustering/#write-consistency
|
|
type Consistency string
|
|
|
|
// BatchSize returns size of batch
|
|
func (o *Options) BatchSize() uint {
|
|
return o.batchSize
|
|
}
|
|
|
|
// SetBatchSize sets number of points sent in single request
|
|
func (o *Options) SetBatchSize(batchSize uint) *Options {
|
|
o.batchSize = batchSize
|
|
return o
|
|
}
|
|
|
|
// FlushInterval returns flush interval in ms
|
|
func (o *Options) FlushInterval() uint {
|
|
return o.flushInterval
|
|
}
|
|
|
|
// SetFlushInterval sets flush interval in ms in which is buffer flushed if it has not been already written
|
|
func (o *Options) SetFlushInterval(flushIntervalMs uint) *Options {
|
|
o.flushInterval = flushIntervalMs
|
|
return o
|
|
}
|
|
|
|
// RetryInterval returns the default retry interval in ms, if not sent by server. Default 5,000.
|
|
func (o *Options) RetryInterval() uint {
|
|
return o.retryInterval
|
|
}
|
|
|
|
// SetRetryInterval sets the time to wait before retry unsuccessful write in ms, if not sent by server
|
|
func (o *Options) SetRetryInterval(retryIntervalMs uint) *Options {
|
|
o.retryInterval = retryIntervalMs
|
|
return o
|
|
}
|
|
|
|
// MaxRetries returns maximum count of retry attempts of failed writes, default 5.
|
|
func (o *Options) MaxRetries() uint {
|
|
return o.maxRetries
|
|
}
|
|
|
|
// SetMaxRetries sets maximum count of retry attempts of failed writes.
|
|
// Setting zero value disables retry strategy.
|
|
func (o *Options) SetMaxRetries(maxRetries uint) *Options {
|
|
o.maxRetries = maxRetries
|
|
return o
|
|
}
|
|
|
|
// RetryBufferLimit returns retry buffer limit.
|
|
func (o *Options) RetryBufferLimit() uint {
|
|
return o.retryBufferLimit
|
|
}
|
|
|
|
// SetRetryBufferLimit sets maximum number of points to keep for retry. Should be multiple of BatchSize.
|
|
func (o *Options) SetRetryBufferLimit(retryBufferLimit uint) *Options {
|
|
o.retryBufferLimit = retryBufferLimit
|
|
return o
|
|
}
|
|
|
|
// MaxRetryInterval returns the maximum delay between each retry attempt in milliseconds, default 125,000.
|
|
func (o *Options) MaxRetryInterval() uint {
|
|
return o.maxRetryInterval
|
|
}
|
|
|
|
// SetMaxRetryInterval sets the maximum delay between each retry attempt in millisecond
|
|
func (o *Options) SetMaxRetryInterval(maxRetryIntervalMs uint) *Options {
|
|
o.maxRetryInterval = maxRetryIntervalMs
|
|
return o
|
|
}
|
|
|
|
// MaxRetryTime returns the maximum total retry timeout in millisecond, default 180,000.
|
|
func (o *Options) MaxRetryTime() uint {
|
|
return o.maxRetryTime
|
|
}
|
|
|
|
// SetMaxRetryTime sets the maximum total retry timeout in millisecond.
|
|
func (o *Options) SetMaxRetryTime(maxRetryTimeMs uint) *Options {
|
|
o.maxRetryTime = maxRetryTimeMs
|
|
return o
|
|
}
|
|
|
|
// ExponentialBase returns the base for the exponential retry delay. Default 2.
|
|
func (o *Options) ExponentialBase() uint {
|
|
return o.exponentialBase
|
|
}
|
|
|
|
// SetExponentialBase sets the base for the exponential retry delay.
|
|
func (o *Options) SetExponentialBase(retryExponentialBase uint) *Options {
|
|
o.exponentialBase = retryExponentialBase
|
|
return o
|
|
}
|
|
|
|
// Precision returns time precision for writes
|
|
func (o *Options) Precision() time.Duration {
|
|
return o.precision
|
|
}
|
|
|
|
// SetPrecision sets time precision to use in writes for timestamp. In unit of duration: time.Nanosecond, time.Microsecond, time.Millisecond, time.Second
|
|
func (o *Options) SetPrecision(precision time.Duration) *Options {
|
|
o.precision = precision
|
|
return o
|
|
}
|
|
|
|
// UseGZip returns true if write request are gzip`ed
|
|
func (o *Options) UseGZip() bool {
|
|
return o.useGZip
|
|
}
|
|
|
|
// SetUseGZip specifies whether to use GZip compression in write requests.
|
|
func (o *Options) SetUseGZip(useGZip bool) *Options {
|
|
o.useGZip = useGZip
|
|
return o
|
|
}
|
|
|
|
// AddDefaultTag adds a default tag. DefaultTags are added to each written point.
|
|
// If a tag with the same key already exist it is overwritten.
|
|
// If a point already defines such a tag, it is left unchanged.
|
|
func (o *Options) AddDefaultTag(key, value string) *Options {
|
|
o.DefaultTags()[key] = value
|
|
return o
|
|
}
|
|
|
|
// DefaultTags returns set of default tags
|
|
func (o *Options) DefaultTags() map[string]string {
|
|
if o.defaultTags == nil {
|
|
o.defaultTags = make(map[string]string)
|
|
}
|
|
return o.defaultTags
|
|
}
|
|
|
|
// Consistency returns consistency for param value
|
|
func (o *Options) Consistency() Consistency {
|
|
return o.consistency
|
|
}
|
|
|
|
// SetConsistency allows setting InfluxDB Enterprise write consistency, as explained in https://docs.influxdata.com/enterprise_influxdb/v1.9/concepts/clustering/#write-consistency */
|
|
func (o *Options) SetConsistency(consistency Consistency) *Options {
|
|
o.consistency = consistency
|
|
return o
|
|
}
|
|
|
|
// DefaultOptions returns Options object with default values
|
|
func DefaultOptions() *Options {
|
|
return &Options{batchSize: 5_000, flushInterval: 1_000, precision: time.Nanosecond, useGZip: false, retryBufferLimit: 50_000, defaultTags: make(map[string]string),
|
|
maxRetries: 5, retryInterval: 5_000, maxRetryInterval: 125_000, maxRetryTime: 180_000, exponentialBase: 2}
|
|
}
|