mirror of
https://github.com/mainflux/mainflux.git
synced 2025-05-04 22:17:59 +08:00

* Upgrade InfluxDB from 1.x to 2.x Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Influx DB configuration updated Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Connection to InfluxDBv2 Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Token cannot be created Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Connected to InfluxDB2 Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Connected to InfluxDB2 Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * InfluxDB v2 Consumer Implementation Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * quickfix Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Influxdb-Writer Unit Tests Update Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Consumer Update Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * InfluxDB Writer Tests Implemented Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * fix Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * InfluxDB Connection Check Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Resolving Remarks Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Resolved consumer-test remark Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * consumer-test slow working version Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * reader changes Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Consumer tests time issue fixed Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Eof warning fixed Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Resolved Reviews Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Makefile Fixed Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Resolved Reviews Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Resolved Reviews Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Resolved Reviews Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Resolved Reviews Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Resolved Reviews Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Readers Initial Setup Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * consumer json fix Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * readers simple version Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * readers simple version fix Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * readers simple version fix Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * readers simple version fix Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * readers simple version fix Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * readers simple version fix Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * readers simple version fix Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Influxdb Reader Parsers Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Influxdb Reader Parsers Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Influxdb Reader Parsers Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Bugfix and resolves comments. Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * one test fails Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * solved last page read Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * writers future time problem fixed Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * weird Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * weird Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Tests Passes Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Resolve Semaphore Issues Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * UUID comment on consumer tests resolved Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Inclusive from and Exclusive to Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Inclusive from and Exclusive to Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * 1 second limits Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * 1 second limits Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * resolved review Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * resolved review Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * resolved review Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * resolved review Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * fixed json time Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * fixed CI error Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * resolved request Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * resolved request Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * resolved requests Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * removed blank line Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * resolved comment Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * deleted unnecessary string builder Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * initial commit Signed-off-by: aryan <aryangodara03@gmail.com> * change influxdb docker image version. Signed-off-by: aryan <aryangodara03@gmail.com> * go mod and vendor fixing Signed-off-by: aryan <aryangodara03@gmail.com> * Upgrade InfluxDB from 1.x to 2.x Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Influx DB configuration updated Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Connection to InfluxDBv2 Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Token cannot be created Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Connected to InfluxDB2 Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Connected to InfluxDB2 Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * InfluxDB v2 Consumer Implementation Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * quickfix Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Influxdb-Writer Unit Tests Update Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Consumer Update Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * InfluxDB Writer Tests Implemented Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * fix Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * InfluxDB Connection Check Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Resolving Remarks Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Resolved consumer-test remark Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * consumer-test slow working version Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * reader changes Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Consumer tests time issue fixed Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Resolved Reviews Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Resolved Reviews Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Resolved Reviews Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Resolved Reviews Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Resolved Reviews Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Resolved Reviews Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Readers Initial Setup Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * consumer json fix Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * readers simple version Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * readers simple version fix Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * readers simple version fix Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * readers simple version fix Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * readers simple version fix Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * readers simple version fix Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * readers simple version fix Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Influxdb Reader Parsers Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Influxdb Reader Parsers Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Influxdb Reader Parsers Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Bugfix and resolves comments. Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * one test fails Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * solved last page read Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * writers future time problem fixed Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * weird Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * weird Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Tests Passes Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Resolve Semaphore Issues Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * UUID comment on consumer tests resolved Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Inclusive from and Exclusive to Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * Inclusive from and Exclusive to Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * 1 second limits Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * 1 second limits Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * resolved review Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * resolved review Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * resolved review Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * resolved review Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * fixed json time Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * fixed CI error Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * resolved request Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * resolved request Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * resolved requests Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * removed blank line Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * resolved comment Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * deleted unnecessary string builder Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> * initial commit Signed-off-by: aryan <aryangodara03@gmail.com> * change influxdb docker image version. Signed-off-by: aryan <aryangodara03@gmail.com> * go mod and vendor fixing Signed-off-by: aryan <aryangodara03@gmail.com> * go mod fixing Signed-off-by: aryan <aryangodara03@gmail.com> * make tests pass locally, fix errors Signed-off-by: aryan <aryangodara03@gmail.com> * rem unsused variables/consts Signed-off-by: aryan <aryangodara03@gmail.com> * go mod tidy vendor Signed-off-by: aryan <aryangodara03@gmail.com> * add env var and data types Signed-off-by: aryan <aryangodara03@gmail.com> * update influxdb version to latest (2.3 to 2.12) Signed-off-by: aryan <aryangodara03@gmail.com> * change time precision for message tests Signed-off-by: aryan <aryangodara03@gmail.com> * renamed influxdb2 to influxdata Signed-off-by: aryan <aryangodara03@gmail.com> * address remarks on PR Signed-off-by: aryan <aryangodara03@gmail.com> * update influxbd according to latest mf commit Signed-off-by: aryan <aryangodara03@gmail.com> * temp commit, rem before pushing Signed-off-by: aryan <aryangodara03@gmail.com> * update main files and cassandra-reader messages_tests Signed-off-by: aryan <aryangodara03@gmail.com> * fix name of logger while importing Signed-off-by: aryan <aryangodara03@gmail.com> * remove unnecessary print lines Signed-off-by: aryan <aryangodara03@gmail.com> * correct env var name Signed-off-by: aryan <aryangodara03@gmail.com> * change to async consume Signed-off-by: aryan <aryangodara03@gmail.com> * add option to switch bw sync and async Signed-off-by: aryan <aryangodara03@gmail.com> * test for both async and sync Signed-off-by: aryan <aryangodara03@gmail.com> * update consumer and add writeAPIs to config Signed-off-by: aryan <aryangodara03@gmail.com> * revert back to sync consuming Signed-off-by: aryan <aryangodara03@gmail.com> * temp fix for default timeout value Signed-off-by: aryan <aryangodara03@gmail.com> * set default timeout in config. Signed-off-by: aryan <aryangodara03@gmail.com> * remove unwanted env vars, add required ones. Signed-off-by: aryan <aryangodara03@gmail.com> * rem unused username password from config Signed-off-by: aryan <aryangodara03@gmail.com> * update readme, env vars, and remove grafana Signed-off-by: aryan <aryangodara03@gmail.com> * update readme Signed-off-by: aryan <aryangodara03@gmail.com> * fix typo Signed-off-by: aryan <aryangodara03@gmail.com> * update readme description. Signed-off-by: aryan <aryangodara03@gmail.com> * fix more typos. Signed-off-by: aryan <aryangodara03@gmail.com> * add link to official docs to readme. Signed-off-by: aryan <aryangodara03@gmail.com> --------- Signed-off-by: fatih <fatihdurmaz@sabanciuniv.edu> Signed-off-by: aryan <aryangodara03@gmail.com> Co-authored-by: fatih <fatihdurmaz@sabanciuniv.edu> Co-authored-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>
193 lines
4.4 KiB
Go
193 lines
4.4 KiB
Go
package protocol
|
|
|
|
import (
|
|
"fmt"
|
|
"io"
|
|
"strings"
|
|
"sync"
|
|
"time"
|
|
)
|
|
|
|
const (
|
|
maxErrorBufferSize = 1024
|
|
)
|
|
|
|
// TimeFunc is used to override the default time for a metric
|
|
// with no specified timestamp.
|
|
type TimeFunc func() time.Time
|
|
|
|
// ParseError indicates a error in the parsing of the text.
|
|
type ParseError struct {
|
|
Offset int
|
|
LineOffset int
|
|
LineNumber int
|
|
Column int
|
|
msg string
|
|
buf string
|
|
}
|
|
|
|
func (e *ParseError) Error() string {
|
|
buffer := e.buf[e.LineOffset:]
|
|
eol := strings.IndexAny(buffer, "\r\n")
|
|
if eol >= 0 {
|
|
buffer = buffer[:eol]
|
|
}
|
|
if len(buffer) > maxErrorBufferSize {
|
|
buffer = buffer[:maxErrorBufferSize] + "..."
|
|
}
|
|
return fmt.Sprintf("metric parse error: %s at %d:%d: %q", e.msg, e.LineNumber, e.Column, buffer)
|
|
}
|
|
|
|
// Parser is an InfluxDB Line Protocol parser that implements the
|
|
// parsers.Parser interface.
|
|
type Parser struct {
|
|
DefaultTags map[string]string
|
|
|
|
sync.Mutex
|
|
*machine
|
|
handler *MetricHandler
|
|
}
|
|
|
|
// NewParser returns a Parser than accepts line protocol
|
|
func NewParser(handler *MetricHandler) *Parser {
|
|
return &Parser{
|
|
machine: NewMachine(handler),
|
|
handler: handler,
|
|
}
|
|
}
|
|
|
|
// NewSeriesParser returns a Parser than accepts a measurement and tagset
|
|
func NewSeriesParser(handler *MetricHandler) *Parser {
|
|
return &Parser{
|
|
machine: NewSeriesMachine(handler),
|
|
handler: handler,
|
|
}
|
|
}
|
|
|
|
// SetTimeFunc allows default times to be set when no time is specified
|
|
// for a metric in line-protocol.
|
|
func (p *Parser) SetTimeFunc(f TimeFunc) {
|
|
p.handler.SetTimeFunc(f)
|
|
}
|
|
|
|
// Parse interprets line-protocol bytes as many metrics.
|
|
func (p *Parser) Parse(input []byte) ([]Metric, error) {
|
|
p.Lock()
|
|
defer p.Unlock()
|
|
metrics := make([]Metric, 0)
|
|
p.machine.SetData(input)
|
|
|
|
for {
|
|
err := p.machine.Next()
|
|
if err == EOF {
|
|
break
|
|
}
|
|
|
|
if err != nil {
|
|
return nil, &ParseError{
|
|
Offset: p.machine.Position(),
|
|
LineOffset: p.machine.LineOffset(),
|
|
LineNumber: p.machine.LineNumber(),
|
|
Column: p.machine.Column(),
|
|
msg: err.Error(),
|
|
buf: string(input),
|
|
}
|
|
}
|
|
|
|
metric, err := p.handler.Metric()
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
|
|
if metric == nil {
|
|
continue
|
|
}
|
|
|
|
metrics = append(metrics, metric)
|
|
}
|
|
|
|
return metrics, nil
|
|
}
|
|
|
|
// StreamParser is an InfluxDB Line Protocol parser. It is not safe for
|
|
// concurrent use in multiple goroutines.
|
|
type StreamParser struct {
|
|
machine *streamMachine
|
|
handler *MetricHandler
|
|
}
|
|
|
|
// NewStreamParser parses from a reader and iterates the machine
|
|
// metric by metric. Not safe for concurrent use in multiple goroutines.
|
|
func NewStreamParser(r io.Reader) *StreamParser {
|
|
handler := NewMetricHandler()
|
|
return &StreamParser{
|
|
machine: NewStreamMachine(r, handler),
|
|
handler: handler,
|
|
}
|
|
}
|
|
|
|
// SetTimeFunc changes the function used to determine the time of metrics
|
|
// without a timestamp. The default TimeFunc is time.Now. Useful mostly for
|
|
// testing, or perhaps if you want all metrics to have the same timestamp.
|
|
func (p *StreamParser) SetTimeFunc(f TimeFunc) {
|
|
p.handler.SetTimeFunc(f)
|
|
}
|
|
|
|
// SetTimePrecision specifies units for the time stamp.
|
|
func (p *StreamParser) SetTimePrecision(u time.Duration) {
|
|
p.handler.SetTimePrecision(u)
|
|
}
|
|
|
|
// Next parses the next item from the stream. You can repeat calls to this
|
|
// function until it returns EOF.
|
|
func (p *StreamParser) Next() (Metric, error) {
|
|
err := p.machine.Next()
|
|
if err == EOF {
|
|
return nil, EOF
|
|
}
|
|
|
|
if err != nil {
|
|
return nil, &ParseError{
|
|
Offset: p.machine.Position(),
|
|
LineOffset: p.machine.LineOffset(),
|
|
LineNumber: p.machine.LineNumber(),
|
|
Column: p.machine.Column(),
|
|
msg: err.Error(),
|
|
buf: p.machine.LineText(),
|
|
}
|
|
}
|
|
|
|
metric, err := p.handler.Metric()
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
|
|
return metric, nil
|
|
}
|
|
|
|
// Position returns the current byte offset into the data.
|
|
func (p *StreamParser) Position() int {
|
|
return p.machine.Position()
|
|
}
|
|
|
|
// LineOffset returns the byte offset of the current line.
|
|
func (p *StreamParser) LineOffset() int {
|
|
return p.machine.LineOffset()
|
|
}
|
|
|
|
// LineNumber returns the current line number. Lines are counted based on the
|
|
// regular expression `\r?\n`.
|
|
func (p *StreamParser) LineNumber() int {
|
|
return p.machine.LineNumber()
|
|
}
|
|
|
|
// Column returns the current column.
|
|
func (p *StreamParser) Column() int {
|
|
return p.machine.Column()
|
|
}
|
|
|
|
// LineText returns the text of the current line that has been parsed so far.
|
|
func (p *StreamParser) LineText() string {
|
|
return p.machine.LineText()
|
|
}
|