1
0
mirror of https://github.com/mainflux/mainflux.git synced 2025-05-01 13:48:56 +08:00
Dušan Borovčanin 3d3aa525a6
NOISSUE - Switch to Google Zanzibar Access control approach (#1919)
* Return Auth service

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

Update Compose to run with SpiceDB and Auth svc

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

Update auth gRPC API

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

Remove Users' policies

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

Move Groups to internal

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

Use shared groups in Users

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

Remove unused code

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

Use pkg Groups in Things

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

Remove Things groups

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

Make imports consistent

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

Update Groups networking

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

Remove things groups-specific API

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

Move Things Clients to the root

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

Move Clients to Users root

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

Temporarily remove tracing

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

Fix imports

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

Add buffer config for gRPC

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

Update auth type for Things

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

Use Auth for login

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

Add temporary solution for refresh token

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

Update Tokenizer interface

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

Updade tokens issuing

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

Fix token issuing

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

Update JWT validator and refactor Tokenizer

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

Rename access timeout

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

Rename login to authenticate

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

Update Identify to use SubjectID

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

Add Auth to Groups

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

Use the Auth service for Groups

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

Update auth schema

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

Fix Auth for Groups

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

Add auth for addons (#14)

Signed-off-by: Arvindh <arvindh91@gmail.com>

Speparate Login and Refresh tokens

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

Merge authN and authZ requests for things

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

Add connect and disconnect

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

Update sharing

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

Fix policies addition and removal

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

Update relation with roels

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

Add gRPC to Things

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

Assign and Unassign members to group and Listing of Group members (#15)

* add auth for addons

Signed-off-by: Arvindh <arvindh91@gmail.com>

* add assign and unassign to group

Signed-off-by: Arvindh <arvindh91@gmail.com>

* add group incomplete repo implementation

Signed-off-by: Arvindh <arvindh91@gmail.com>

* groups for users

Signed-off-by: Arvindh <arvindh91@gmail.com>

* groups for users

Signed-off-by: Arvindh <arvindh91@gmail.com>

* groups for users

Signed-off-by: Arvindh <arvindh91@gmail.com>

* groups for users

Signed-off-by: Arvindh <arvindh91@gmail.com>

---------

Signed-off-by: Arvindh <arvindh91@gmail.com>

Move coap mqtt and ws policies to spicedb (#16)

Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>

Remove old policies

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

NOISSUE - Things authorize to return thingID (#18)

This commit modifies the authorize endpoint to the grpc endpoint to return thingID. The authorize endpoint allows adapters to get the publisher of the message.

Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>

Add Groups to users service (#17)

* add assign and unassign to group

Signed-off-by: Arvindh <arvindh91@gmail.com>

* add group incomplete repo implementation

Signed-off-by: Arvindh <arvindh91@gmail.com>

* groups for users

Signed-off-by: Arvindh <arvindh91@gmail.com>

* groups for users

Signed-off-by: Arvindh <arvindh91@gmail.com>

* groups for users

Signed-off-by: Arvindh <arvindh91@gmail.com>

* groups for users

Signed-off-by: Arvindh <arvindh91@gmail.com>

* groups for users stable 1

Signed-off-by: Arvindh <arvindh91@gmail.com>

* groups for users stable 2

Signed-off-by: Arvindh <arvindh91@gmail.com>

* groups for users & things

Signed-off-by: Arvindh <arvindh91@gmail.com>

* Amend signature

Signed-off-by: Arvindh <arvindh91@gmail.com>

* fix merge error

Signed-off-by: Arvindh <arvindh91@gmail.com>

---------

Signed-off-by: Arvindh <arvindh91@gmail.com>
Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

* NOISSUE - Fix es code (#21)

Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>
Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

* NOISSUE - Fix Bugs (#20)

* fix bugs

Signed-off-by: Arvindh <arvindh91@gmail.com>

* fix bugs

Signed-off-by: Arvindh <arvindh91@gmail.com>

---------

Signed-off-by: Arvindh <arvindh91@gmail.com>
Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

* NOISSUE - Test e2e (#19)

* fix: connect method

Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>

* fix: e2e

Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>

* fix changes in sdk and e2e

Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>

* feat(docker): remove unnecessary port mapping

Remove the port mapping for MQTT broker in the docker-compose.yml file.

Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>

* Enable group listing

Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>

* feat(responses): update ChannelsPage struct

The ChannelsPage struct in the responses.go file has been updated. The "Channels" field has been renamed to "Groups" to provide more accurate naming. This change ensures consistency and clarity in the codebase.

Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>

* feat(things): add UpdateClientSecret method

Add the UpdateClientSecret method to the things service. This method allows updating the client secret for a specific client identified by the provided token, id, and key parameters.

Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>

---------

Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>
Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

* Use smaller buffers for gRPC

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

* Clean up tests (#22)

Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>
Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

* Add Connect Disconnect endpoints (#23)

* fix bugs

Signed-off-by: Arvindh <arvindh91@gmail.com>

* fix bugs

Signed-off-by: Arvindh <arvindh91@gmail.com>

* fix list of things in a channel and Add connect disconnect endpoint

Signed-off-by: Arvindh <arvindh91@gmail.com>

* fix list of things in a channel and Add connect disconnect endpoint

Signed-off-by: Arvindh <arvindh91@gmail.com>

---------

Signed-off-by: Arvindh <arvindh91@gmail.com>
Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

* Add: Things share with users (#25)

* fix list of things in a channel and Add connect disconnect endpoint

Signed-off-by: Arvindh <arvindh91@gmail.com>

* add: things share with other users

Signed-off-by: Arvindh <arvindh91@gmail.com>

---------

Signed-off-by: Arvindh <arvindh91@gmail.com>
Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

* NOISSUE - Rename gRPC Services (#24)

* Rename things and users auth service

Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>

* docs: add authorization docs for gRPC services

Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>

* Rename things and users grpc services

Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>

* Remove mainflux.env package

Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>

---------

Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>
Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

* Add: Listing of things, channels, groups, users  (#26)

* add: listing of channels, users, groups, things

Signed-off-by: Arvindh <arvindh91@gmail.com>

* add: listing of channels, users, groups, things

Signed-off-by: Arvindh <arvindh91@gmail.com>

* add: listing of channels, users, groups, things

Signed-off-by: Arvindh <arvindh91@gmail.com>

* add: listing of channels, users, groups, things

Signed-off-by: Arvindh <arvindh91@gmail.com>

---------

Signed-off-by: Arvindh <arvindh91@gmail.com>
Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

* NOISSUE - Clean Up Users (#27)

* feat(groups): rename redis package to events

- Renamed the `redis` package to `events` in the `internal/groups` directory.
- Updated the file paths and names accordingly.
- This change reflects the more accurate purpose of the package and improves code organization.

Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>

* feat(auth): Modify identity method

Change request and response of identity method

Add accessToken and refreshToken to Token response

Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>

* clean up users, remove dead code

Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>

* feat(users): add unit tests for user service

This commit adds unit tests for the user service in the `users` package. The tests cover various scenarios and ensure the correct behavior of the service.

Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>

---------

Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>
Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

* Add: List of user groups & removed repeating code in groups (#29)

* removed repeating code in list groups

Signed-off-by: Arvindh <arvindh91@gmail.com>

* add: list of user group

Signed-off-by: Arvindh <arvindh91@gmail.com>

* fix: otel handler operator name for endpoints

Signed-off-by: Arvindh <arvindh91@gmail.com>

---------

Signed-off-by: Arvindh <arvindh91@gmail.com>
Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

* NOISSUE - Clean Up Things Service (#28)

* Rework things service

Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>

* add tests

Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>

---------

Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>
Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

* NOISSUE - Clean Up Auth Service (#30)

* clean up auth service

Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>

* feat(auth): remove unused import

Remove the unused import of `emptypb` in `auth.pb.go`. This import is not being used in the codebase and can be safely removed.

Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>

---------

Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>
Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

* NOISSUE - Update API docs (#31)

Signed-off-by: rodneyosodo <blackd0t@protonmail.com>
Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

* Remove TODO comments and cleanup the code

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

* Update dependenices

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

---------

Signed-off-by: Arvindh <arvindh91@gmail.com>
Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>
Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>
Co-authored-by: b1ackd0t <28790446+rodneyosodo@users.noreply.github.com>
Co-authored-by: Arvindh <30824765+arvindh123@users.noreply.github.com>
2023-10-15 22:02:13 +02:00

723 lines
23 KiB
Go

// Copyright (C) MongoDB, Inc. 2017-present.
//
// Licensed under the Apache License, Version 2.0 (the "License"); you may
// not use this file except in compliance with the License. You may obtain
// a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
package mongo
import (
"context"
"errors"
"fmt"
"reflect"
"strconv"
"time"
"go.mongodb.org/mongo-driver/bson"
"go.mongodb.org/mongo-driver/bson/bsoncodec"
"go.mongodb.org/mongo-driver/bson/primitive"
"go.mongodb.org/mongo-driver/internal"
"go.mongodb.org/mongo-driver/mongo/description"
"go.mongodb.org/mongo-driver/mongo/options"
"go.mongodb.org/mongo-driver/mongo/readconcern"
"go.mongodb.org/mongo-driver/mongo/readpref"
"go.mongodb.org/mongo-driver/x/bsonx/bsoncore"
"go.mongodb.org/mongo-driver/x/mongo/driver"
"go.mongodb.org/mongo-driver/x/mongo/driver/operation"
"go.mongodb.org/mongo-driver/x/mongo/driver/session"
)
var (
// ErrMissingResumeToken indicates that a change stream notification from the server did not contain a resume token.
ErrMissingResumeToken = errors.New("cannot provide resume functionality when the resume token is missing")
// ErrNilCursor indicates that the underlying cursor for the change stream is nil.
ErrNilCursor = errors.New("cursor is nil")
minResumableLabelWireVersion int32 = 9 // Wire version at which the server includes the resumable error label
networkErrorLabel = "NetworkError"
resumableErrorLabel = "ResumableChangeStreamError"
errorCursorNotFound int32 = 43 // CursorNotFound error code
// Allowlist of error codes that are considered resumable.
resumableChangeStreamErrors = map[int32]struct{}{
6: {}, // HostUnreachable
7: {}, // HostNotFound
89: {}, // NetworkTimeout
91: {}, // ShutdownInProgress
189: {}, // PrimarySteppedDown
262: {}, // ExceededTimeLimit
9001: {}, // SocketException
10107: {}, // NotPrimary
11600: {}, // InterruptedAtShutdown
11602: {}, // InterruptedDueToReplStateChange
13435: {}, // NotPrimaryNoSecondaryOK
13436: {}, // NotPrimaryOrSecondary
63: {}, // StaleShardVersion
150: {}, // StaleEpoch
13388: {}, // StaleConfig
234: {}, // RetryChangeStream
133: {}, // FailedToSatisfyReadPreference
}
)
// ChangeStream is used to iterate over a stream of events. Each event can be decoded into a Go type via the Decode
// method or accessed as raw BSON via the Current field. This type is not goroutine safe and must not be used
// concurrently by multiple goroutines. For more information about change streams, see
// https://www.mongodb.com/docs/manual/changeStreams/.
type ChangeStream struct {
// Current is the BSON bytes of the current event. This property is only valid until the next call to Next or
// TryNext. If continued access is required, a copy must be made.
Current bson.Raw
aggregate *operation.Aggregate
pipelineSlice []bsoncore.Document
pipelineOptions map[string]bsoncore.Value
cursor changeStreamCursor
cursorOptions driver.CursorOptions
batch []bsoncore.Document
resumeToken bson.Raw
err error
sess *session.Client
client *Client
bsonOpts *options.BSONOptions
registry *bsoncodec.Registry
streamType StreamType
options *options.ChangeStreamOptions
selector description.ServerSelector
operationTime *primitive.Timestamp
wireVersion *description.VersionRange
}
type changeStreamConfig struct {
readConcern *readconcern.ReadConcern
readPreference *readpref.ReadPref
client *Client
bsonOpts *options.BSONOptions
registry *bsoncodec.Registry
streamType StreamType
collectionName string
databaseName string
crypt driver.Crypt
}
func newChangeStream(ctx context.Context, config changeStreamConfig, pipeline interface{},
opts ...*options.ChangeStreamOptions) (*ChangeStream, error) {
if ctx == nil {
ctx = context.Background()
}
cs := &ChangeStream{
client: config.client,
bsonOpts: config.bsonOpts,
registry: config.registry,
streamType: config.streamType,
options: options.MergeChangeStreamOptions(opts...),
selector: description.CompositeSelector([]description.ServerSelector{
description.ReadPrefSelector(config.readPreference),
description.LatencySelector(config.client.localThreshold),
}),
cursorOptions: config.client.createBaseCursorOptions(),
}
cs.sess = sessionFromContext(ctx)
if cs.sess == nil && cs.client.sessionPool != nil {
cs.sess = session.NewImplicitClientSession(cs.client.sessionPool, cs.client.id)
}
if cs.err = cs.client.validSession(cs.sess); cs.err != nil {
closeImplicitSession(cs.sess)
return nil, cs.Err()
}
cs.aggregate = operation.NewAggregate(nil).
ReadPreference(config.readPreference).ReadConcern(config.readConcern).
Deployment(cs.client.deployment).ClusterClock(cs.client.clock).
CommandMonitor(cs.client.monitor).Session(cs.sess).ServerSelector(cs.selector).Retry(driver.RetryNone).
ServerAPI(cs.client.serverAPI).Crypt(config.crypt).Timeout(cs.client.timeout)
if cs.options.Collation != nil {
cs.aggregate.Collation(bsoncore.Document(cs.options.Collation.ToDocument()))
}
if comment := cs.options.Comment; comment != nil {
cs.aggregate.Comment(*comment)
commentVal, err := marshalValue(comment, cs.bsonOpts, cs.registry)
if err != nil {
return nil, err
}
cs.cursorOptions.Comment = commentVal
}
if cs.options.BatchSize != nil {
cs.aggregate.BatchSize(*cs.options.BatchSize)
cs.cursorOptions.BatchSize = *cs.options.BatchSize
}
if cs.options.MaxAwaitTime != nil {
cs.cursorOptions.MaxTimeMS = int64(*cs.options.MaxAwaitTime / time.Millisecond)
}
if cs.options.Custom != nil {
// Marshal all custom options before passing to the initial aggregate. Return
// any errors from Marshaling.
customOptions := make(map[string]bsoncore.Value)
for optionName, optionValue := range cs.options.Custom {
bsonType, bsonData, err := bson.MarshalValueWithRegistry(cs.registry, optionValue)
if err != nil {
cs.err = err
closeImplicitSession(cs.sess)
return nil, cs.Err()
}
optionValueBSON := bsoncore.Value{Type: bsonType, Data: bsonData}
customOptions[optionName] = optionValueBSON
}
cs.aggregate.CustomOptions(customOptions)
}
if cs.options.CustomPipeline != nil {
// Marshal all custom pipeline options before building pipeline slice. Return
// any errors from Marshaling.
cs.pipelineOptions = make(map[string]bsoncore.Value)
for optionName, optionValue := range cs.options.CustomPipeline {
bsonType, bsonData, err := bson.MarshalValueWithRegistry(cs.registry, optionValue)
if err != nil {
cs.err = err
closeImplicitSession(cs.sess)
return nil, cs.Err()
}
optionValueBSON := bsoncore.Value{Type: bsonType, Data: bsonData}
cs.pipelineOptions[optionName] = optionValueBSON
}
}
switch cs.streamType {
case ClientStream:
cs.aggregate.Database("admin")
case DatabaseStream:
cs.aggregate.Database(config.databaseName)
case CollectionStream:
cs.aggregate.Collection(config.collectionName).Database(config.databaseName)
default:
closeImplicitSession(cs.sess)
return nil, fmt.Errorf("must supply a valid StreamType in config, instead of %v", cs.streamType)
}
// When starting a change stream, cache startAfter as the first resume token if it is set. If not, cache
// resumeAfter. If neither is set, do not cache a resume token.
resumeToken := cs.options.StartAfter
if resumeToken == nil {
resumeToken = cs.options.ResumeAfter
}
var marshaledToken bson.Raw
if resumeToken != nil {
if marshaledToken, cs.err = bson.Marshal(resumeToken); cs.err != nil {
closeImplicitSession(cs.sess)
return nil, cs.Err()
}
}
cs.resumeToken = marshaledToken
if cs.err = cs.buildPipelineSlice(pipeline); cs.err != nil {
closeImplicitSession(cs.sess)
return nil, cs.Err()
}
var pipelineArr bsoncore.Document
pipelineArr, cs.err = cs.pipelineToBSON()
cs.aggregate.Pipeline(pipelineArr)
if cs.err = cs.executeOperation(ctx, false); cs.err != nil {
closeImplicitSession(cs.sess)
return nil, cs.Err()
}
return cs, cs.Err()
}
func (cs *ChangeStream) createOperationDeployment(server driver.Server, connection driver.Connection) driver.Deployment {
return &changeStreamDeployment{
topologyKind: cs.client.deployment.Kind(),
server: server,
conn: connection,
}
}
func (cs *ChangeStream) executeOperation(ctx context.Context, resuming bool) error {
var server driver.Server
var conn driver.Connection
if server, cs.err = cs.client.deployment.SelectServer(ctx, cs.selector); cs.err != nil {
return cs.Err()
}
if conn, cs.err = server.Connection(ctx); cs.err != nil {
return cs.Err()
}
defer conn.Close()
cs.wireVersion = conn.Description().WireVersion
cs.aggregate.Deployment(cs.createOperationDeployment(server, conn))
if resuming {
cs.replaceOptions(cs.wireVersion)
csOptDoc, err := cs.createPipelineOptionsDoc()
if err != nil {
return err
}
pipIdx, pipDoc := bsoncore.AppendDocumentStart(nil)
pipDoc = bsoncore.AppendDocumentElement(pipDoc, "$changeStream", csOptDoc)
if pipDoc, cs.err = bsoncore.AppendDocumentEnd(pipDoc, pipIdx); cs.err != nil {
return cs.Err()
}
cs.pipelineSlice[0] = pipDoc
var plArr bsoncore.Document
if plArr, cs.err = cs.pipelineToBSON(); cs.err != nil {
return cs.Err()
}
cs.aggregate.Pipeline(plArr)
}
// If no deadline is set on the passed-in context, cs.client.timeout is set, and context is not already
// a Timeout context, honor cs.client.timeout in new Timeout context for change stream operation execution
// and potential retry.
if _, deadlineSet := ctx.Deadline(); !deadlineSet && cs.client.timeout != nil && !internal.IsTimeoutContext(ctx) {
newCtx, cancelFunc := internal.MakeTimeoutContext(ctx, *cs.client.timeout)
// Redefine ctx to be the new timeout-derived context.
ctx = newCtx
// Cancel the timeout-derived context at the end of executeOperation to avoid a context leak.
defer cancelFunc()
}
// Execute the aggregate, retrying on retryable errors once (1) if retryable reads are enabled and
// infinitely (-1) if context is a Timeout context.
var retries int
if cs.client.retryReads {
retries = 1
}
if internal.IsTimeoutContext(ctx) {
retries = -1
}
var err error
AggregateExecuteLoop:
for {
err = cs.aggregate.Execute(ctx)
// If no error or no retries remain, do not retry.
if err == nil || retries == 0 {
break AggregateExecuteLoop
}
switch tt := err.(type) {
case driver.Error:
// If error is not retryable, do not retry.
if !tt.RetryableRead() {
break AggregateExecuteLoop
}
// If error is retryable: subtract 1 from retries, redo server selection, checkout
// a connection, and restart loop.
retries--
server, err = cs.client.deployment.SelectServer(ctx, cs.selector)
if err != nil {
break AggregateExecuteLoop
}
conn.Close()
conn, err = server.Connection(ctx)
if err != nil {
break AggregateExecuteLoop
}
defer conn.Close()
// Update the wire version with data from the new connection.
cs.wireVersion = conn.Description().WireVersion
// Reset deployment.
cs.aggregate.Deployment(cs.createOperationDeployment(server, conn))
default:
// Do not retry if error is not a driver error.
break AggregateExecuteLoop
}
}
if err != nil {
cs.err = replaceErrors(err)
return cs.err
}
cr := cs.aggregate.ResultCursorResponse()
cr.Server = server
cs.cursor, cs.err = driver.NewBatchCursor(cr, cs.sess, cs.client.clock, cs.cursorOptions)
if cs.err = replaceErrors(cs.err); cs.err != nil {
return cs.Err()
}
cs.updatePbrtFromCommand()
if cs.options.StartAtOperationTime == nil && cs.options.ResumeAfter == nil &&
cs.options.StartAfter == nil && cs.wireVersion.Max >= 7 &&
cs.emptyBatch() && cs.resumeToken == nil {
cs.operationTime = cs.sess.OperationTime
}
return cs.Err()
}
// Updates the post batch resume token after a successful aggregate or getMore operation.
func (cs *ChangeStream) updatePbrtFromCommand() {
// Only cache the pbrt if an empty batch was returned and a pbrt was included
if pbrt := cs.cursor.PostBatchResumeToken(); cs.emptyBatch() && pbrt != nil {
cs.resumeToken = bson.Raw(pbrt)
}
}
func (cs *ChangeStream) storeResumeToken() error {
// If cs.Current is the last document in the batch and a pbrt is included, cache the pbrt
// Otherwise, cache the _id of the document
var tokenDoc bson.Raw
if len(cs.batch) == 0 {
if pbrt := cs.cursor.PostBatchResumeToken(); pbrt != nil {
tokenDoc = bson.Raw(pbrt)
}
}
if tokenDoc == nil {
var ok bool
tokenDoc, ok = cs.Current.Lookup("_id").DocumentOK()
if !ok {
_ = cs.Close(context.Background())
return ErrMissingResumeToken
}
}
cs.resumeToken = tokenDoc
return nil
}
func (cs *ChangeStream) buildPipelineSlice(pipeline interface{}) error {
val := reflect.ValueOf(pipeline)
if !val.IsValid() || !(val.Kind() == reflect.Slice) {
cs.err = errors.New("can only marshal slices and arrays into aggregation pipelines, but got invalid")
return cs.err
}
cs.pipelineSlice = make([]bsoncore.Document, 0, val.Len()+1)
csIdx, csDoc := bsoncore.AppendDocumentStart(nil)
csDocTemp, err := cs.createPipelineOptionsDoc()
if err != nil {
return err
}
csDoc = bsoncore.AppendDocumentElement(csDoc, "$changeStream", csDocTemp)
csDoc, cs.err = bsoncore.AppendDocumentEnd(csDoc, csIdx)
if cs.err != nil {
return cs.err
}
cs.pipelineSlice = append(cs.pipelineSlice, csDoc)
for i := 0; i < val.Len(); i++ {
var elem []byte
elem, cs.err = marshal(val.Index(i).Interface(), cs.bsonOpts, cs.registry)
if cs.err != nil {
return cs.err
}
cs.pipelineSlice = append(cs.pipelineSlice, elem)
}
return cs.err
}
func (cs *ChangeStream) createPipelineOptionsDoc() (bsoncore.Document, error) {
plDocIdx, plDoc := bsoncore.AppendDocumentStart(nil)
if cs.streamType == ClientStream {
plDoc = bsoncore.AppendBooleanElement(plDoc, "allChangesForCluster", true)
}
if cs.options.FullDocument != nil && *cs.options.FullDocument != options.Default {
plDoc = bsoncore.AppendStringElement(plDoc, "fullDocument", string(*cs.options.FullDocument))
}
if cs.options.FullDocumentBeforeChange != nil {
plDoc = bsoncore.AppendStringElement(plDoc, "fullDocumentBeforeChange", string(*cs.options.FullDocumentBeforeChange))
}
if cs.options.ResumeAfter != nil {
var raDoc bsoncore.Document
raDoc, cs.err = marshal(cs.options.ResumeAfter, cs.bsonOpts, cs.registry)
if cs.err != nil {
return nil, cs.err
}
plDoc = bsoncore.AppendDocumentElement(plDoc, "resumeAfter", raDoc)
}
if cs.options.ShowExpandedEvents != nil {
plDoc = bsoncore.AppendBooleanElement(plDoc, "showExpandedEvents", *cs.options.ShowExpandedEvents)
}
if cs.options.StartAfter != nil {
var saDoc bsoncore.Document
saDoc, cs.err = marshal(cs.options.StartAfter, cs.bsonOpts, cs.registry)
if cs.err != nil {
return nil, cs.err
}
plDoc = bsoncore.AppendDocumentElement(plDoc, "startAfter", saDoc)
}
if cs.options.StartAtOperationTime != nil {
plDoc = bsoncore.AppendTimestampElement(plDoc, "startAtOperationTime", cs.options.StartAtOperationTime.T, cs.options.StartAtOperationTime.I)
}
// Append custom pipeline options.
for optionName, optionValue := range cs.pipelineOptions {
plDoc = bsoncore.AppendValueElement(plDoc, optionName, optionValue)
}
if plDoc, cs.err = bsoncore.AppendDocumentEnd(plDoc, plDocIdx); cs.err != nil {
return nil, cs.err
}
return plDoc, nil
}
func (cs *ChangeStream) pipelineToBSON() (bsoncore.Document, error) {
pipelineDocIdx, pipelineArr := bsoncore.AppendArrayStart(nil)
for i, doc := range cs.pipelineSlice {
pipelineArr = bsoncore.AppendDocumentElement(pipelineArr, strconv.Itoa(i), doc)
}
if pipelineArr, cs.err = bsoncore.AppendArrayEnd(pipelineArr, pipelineDocIdx); cs.err != nil {
return nil, cs.err
}
return pipelineArr, cs.err
}
func (cs *ChangeStream) replaceOptions(wireVersion *description.VersionRange) {
// Cached resume token: use the resume token as the resumeAfter option and set no other resume options
if cs.resumeToken != nil {
cs.options.SetResumeAfter(cs.resumeToken)
cs.options.SetStartAfter(nil)
cs.options.SetStartAtOperationTime(nil)
return
}
// No cached resume token but cached operation time: use the operation time as the startAtOperationTime option and
// set no other resume options
if (cs.sess.OperationTime != nil || cs.options.StartAtOperationTime != nil) && wireVersion.Max >= 7 {
opTime := cs.options.StartAtOperationTime
if cs.operationTime != nil {
opTime = cs.sess.OperationTime
}
cs.options.SetStartAtOperationTime(opTime)
cs.options.SetResumeAfter(nil)
cs.options.SetStartAfter(nil)
return
}
// No cached resume token or operation time: set none of the resume options
cs.options.SetResumeAfter(nil)
cs.options.SetStartAfter(nil)
cs.options.SetStartAtOperationTime(nil)
}
// ID returns the ID for this change stream, or 0 if the cursor has been closed or exhausted.
func (cs *ChangeStream) ID() int64 {
if cs.cursor == nil {
return 0
}
return cs.cursor.ID()
}
// SetBatchSize sets the number of documents to fetch from the database with
// each iteration of the ChangeStream's "Next" or "TryNext" method. This setting
// only affects subsequent document batches fetched from the database.
func (cs *ChangeStream) SetBatchSize(size int32) {
// Set batch size on the cursor options also so any "resumed" change stream
// cursors will pick up the latest batch size setting.
cs.cursorOptions.BatchSize = size
cs.cursor.SetBatchSize(size)
}
// Decode will unmarshal the current event document into val and return any errors from the unmarshalling process
// without any modification. If val is nil or is a typed nil, an error will be returned.
func (cs *ChangeStream) Decode(val interface{}) error {
if cs.cursor == nil {
return ErrNilCursor
}
dec, err := getDecoder(cs.Current, cs.bsonOpts, cs.registry)
if err != nil {
return fmt.Errorf("error configuring BSON decoder: %w", err)
}
return dec.Decode(val)
}
// Err returns the last error seen by the change stream, or nil if no errors has occurred.
func (cs *ChangeStream) Err() error {
if cs.err != nil {
return replaceErrors(cs.err)
}
if cs.cursor == nil {
return nil
}
return replaceErrors(cs.cursor.Err())
}
// Close closes this change stream and the underlying cursor. Next and TryNext must not be called after Close has been
// called. Close is idempotent. After the first call, any subsequent calls will not change the state.
func (cs *ChangeStream) Close(ctx context.Context) error {
if ctx == nil {
ctx = context.Background()
}
defer closeImplicitSession(cs.sess)
if cs.cursor == nil {
return nil // cursor is already closed
}
cs.err = replaceErrors(cs.cursor.Close(ctx))
cs.cursor = nil
return cs.Err()
}
// ResumeToken returns the last cached resume token for this change stream, or nil if a resume token has not been
// stored.
func (cs *ChangeStream) ResumeToken() bson.Raw {
return cs.resumeToken
}
// Next gets the next event for this change stream. It returns true if there were no errors and the next event document
// is available.
//
// Next blocks until an event is available, an error occurs, or ctx expires. If ctx expires, the error
// will be set to ctx.Err(). In an error case, Next will return false.
//
// If Next returns false, subsequent calls will also return false.
func (cs *ChangeStream) Next(ctx context.Context) bool {
return cs.next(ctx, false)
}
// TryNext attempts to get the next event for this change stream. It returns true if there were no errors and the next
// event document is available.
//
// TryNext returns false if the change stream is closed by the server, an error occurs when getting changes from the
// server, the next change is not yet available, or ctx expires. If ctx expires, the error will be set to ctx.Err().
//
// If TryNext returns false and an error occurred or the change stream was closed
// (i.e. cs.Err() != nil || cs.ID() == 0), subsequent attempts will also return false. Otherwise, it is safe to call
// TryNext again until a change is available.
//
// This method requires driver version >= 1.2.0.
func (cs *ChangeStream) TryNext(ctx context.Context) bool {
return cs.next(ctx, true)
}
func (cs *ChangeStream) next(ctx context.Context, nonBlocking bool) bool {
// return false right away if the change stream has already errored or if cursor is closed.
if cs.err != nil {
return false
}
if ctx == nil {
ctx = context.Background()
}
if len(cs.batch) == 0 {
cs.loopNext(ctx, nonBlocking)
if cs.err != nil {
cs.err = replaceErrors(cs.err)
return false
}
if len(cs.batch) == 0 {
return false
}
}
// successfully got non-empty batch
cs.Current = bson.Raw(cs.batch[0])
cs.batch = cs.batch[1:]
if cs.err = cs.storeResumeToken(); cs.err != nil {
return false
}
return true
}
func (cs *ChangeStream) loopNext(ctx context.Context, nonBlocking bool) {
for {
if cs.cursor == nil {
return
}
if cs.cursor.Next(ctx) {
// non-empty batch returned
cs.batch, cs.err = cs.cursor.Batch().Documents()
return
}
cs.err = replaceErrors(cs.cursor.Err())
if cs.err == nil {
// Check if cursor is alive
if cs.ID() == 0 {
return
}
// If a getMore was done but the batch was empty, the batch cursor will return false with no error.
// Update the tracked resume token to catch the post batch resume token from the server response.
cs.updatePbrtFromCommand()
if nonBlocking {
// stop after a successful getMore, even though the batch was empty
return
}
continue // loop getMore until a non-empty batch is returned or an error occurs
}
if !cs.isResumableError() {
return
}
// ignore error from cursor close because if the cursor is deleted or errors we tried to close it and will remake and try to get next batch
_ = cs.cursor.Close(ctx)
if cs.err = cs.executeOperation(ctx, true); cs.err != nil {
return
}
}
}
func (cs *ChangeStream) isResumableError() bool {
commandErr, ok := cs.err.(CommandError)
if !ok || commandErr.HasErrorLabel(networkErrorLabel) {
// All non-server errors or network errors are resumable.
return true
}
if commandErr.Code == errorCursorNotFound {
return true
}
// For wire versions 9 and above, a server error is resumable if it has the ResumableChangeStreamError label.
if cs.wireVersion != nil && cs.wireVersion.Includes(minResumableLabelWireVersion) {
return commandErr.HasErrorLabel(resumableErrorLabel)
}
// For wire versions below 9, a server error is resumable if its code is on the allowlist.
_, resumable := resumableChangeStreamErrors[commandErr.Code]
return resumable
}
// Returns true if the underlying cursor's batch is empty
func (cs *ChangeStream) emptyBatch() bool {
return cs.cursor.Batch().Empty()
}
// StreamType represents the cluster type against which a ChangeStream was created.
type StreamType uint8
// These constants represent valid change stream types. A change stream can be initialized over a collection, all
// collections in a database, or over a cluster.
const (
CollectionStream StreamType = iota
DatabaseStream
ClientStream
)