feat(sources): add source latency metric

[gwenaskell]

Summary

This PR aims at generalizing the metric http_server_handler_duration_seconds by producing metrics giving insight about events batches handling time to all sources, in order to help debugging latency and backpressure issues.

It adds two distributions, source_send_latency_seconds and source_send_batch_latency_seconds, recording the time spent waiting for a single chunk/event, and for a full batch, to be pulled by the buffer.

For the sake of simplicity, those metrics are directly added to the source_sender's Output object so that they will be transparently added to all existing sources. This means that the metrics do not account for the processing time by the source itself (decoding and enriching the events), neither the time elapsed before returning an ack. The former is usually difficult to record without complex refactoring of the source implementation because decoding is buried in the source reader channel, but the computing time used by the source itself is usually small in comparison to the delay caused by downstream latency. The latter however (acks) might be significant if using E2E acks, and since not all sources support acknowledgments (and those that do have a different way of implementing it), it will probably require case-by-case implementation.

Vector configuration

n/a

How did you test this PR?

Change Type

• Bug fix
• New feature
• Dependencies
• Non-functional (chore, refactoring, docs)
• Performance

Is this a breaking change?

• Yes
• No

Does this PR include user facing changes?

• Yes. Please add a changelog fragment based on our guidelines.
• No. A maintainer will apply the no-changelog label to this PR.

References

• Closes: OPA-4855

Notes

• Please read our Vector contributor resources.
• Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
• Some CI checks run only after we manually approve them.
  • We recommend adding a pre-push hook, please see this template.
  • Alternatively, we recommend running the following locally before pushing to the remote branch:
    • make fmt
    • make check-clippy (if there are failures it's possible some of them can be fixed with make clippy-fix)
    • make test
• After a review is requested, please avoid force pushes to help us review incrementally.
  • Feel free to push as many commits as you want. They will be squashed into one before merging.
  • For example, you can run git merge origin master and git push.
• If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
  run make build-licenses to regenerate the license inventory and commit the changes (if any). More details here.

[bruceg]

I think this is the same as send_reference.elapsed()