Hi,
In trying to figure out how the snapshot service is working (in
practice) I've collected some metrics from ftpsync logs on sallinen.
Since 2024-01-26 (the earliest ftpsync logs found) we've finished 15
ftpsyncs + snapshot imports (import + index). The currently running
snapshot import is running since yesterday afternoon (15:22 UTC). I'm
assuming the dates in ftpsync logs are UTC.
Inserted below are start (datetime), end (datetime) and duration
(seconds) for rsync stages 1 and 2 ("R1" and "R2") and snapshot imports
("SN"). I've attached the script used too, for the curious or sceptical.
rsync times vary between 28 and 75 minutes for stage1 and between 4 and
13 minutes for stage2.
snapshot import times vary between 83 minutes and 39 hours. All February
runs are worse than any January run.
--8<---------------cut here---------------start------------->8---
$ ls -rt logs/sallinen/archvsync/ftpsync.log* | xargs cat | ./grok-logs-ftpsync
R1 2024-01-26T02:25:40Z 2024-01-26T02:55:13Z 1773
R2 2024-01-26T02:55:13Z 2024-01-26T03:00:36Z 323
SN 2024-01-26T03:00:37Z 2024-01-26T20:16:46Z 62169
R1 2024-01-26T20:30:33Z 2024-01-26T21:06:44Z 2171
R2 2024-01-26T21:06:44Z 2024-01-26T21:11:00Z 256
SN 2024-01-26T21:11:01Z 2024-01-27T12:25:35Z 54874
R1 2024-01-27T14:36:12Z 2024-01-27T15:17:16Z 2464
R2 2024-01-27T15:17:16Z 2024-01-27T15:22:01Z 285
SN 2024-01-27T15:22:02Z 2024-01-28T03:00:20Z 41898
R1 2024-01-28T08:28:08Z 2024-01-28T09:07:52Z 2384
R2 2024-01-28T09:07:52Z 2024-01-28T09:12:38Z 286
SN 2024-01-28T09:12:38Z 2024-01-29T02:33:55Z 62477
R1 2024-01-29T08:28:37Z 2024-01-29T09:06:32Z 2275
R2 2024-01-29T09:06:32Z 2024-01-29T09:10:20Z 228
SN 2024-01-29T09:10:21Z 2024-01-30T02:38:35Z 62894
R1 2024-01-30T08:28:53Z 2024-01-30T09:03:40Z 2087
R2 2024-01-30T09:03:40Z 2024-01-30T09:08:17Z 277
SN 2024-01-30T09:08:18Z 2024-01-30T11:39:04Z 9046
R1 2024-01-30T14:47:00Z 2024-01-30T15:22:36Z 2136
R2 2024-01-30T15:22:36Z 2024-01-30T15:28:09Z 333
SN 2024-01-30T15:28:10Z 2024-01-30T16:51:36Z 5006
R1 2024-01-30T20:45:28Z 2024-01-30T21:13:25Z 1677
R2 2024-01-30T21:13:25Z 2024-01-30T21:25:53Z 748
SN 2024-01-30T21:25:53Z 2024-01-31T05:09:12Z 27799
R1 2024-01-31T14:52:19Z 2024-01-31T15:31:46Z 2367
R2 2024-01-31T15:31:46Z 2024-01-31T15:38:24Z 398
SN 2024-01-31T15:38:25Z 2024-02-01T07:04:09Z 55544
R1 2024-02-01T08:38:35Z 2024-02-01T09:30:45Z 3130
R2 2024-02-01T09:30:45Z 2024-02-01T09:35:19Z 274
SN 2024-02-01T09:35:19Z 2024-02-02T06:54:50Z 76771
R1 2024-02-02T08:50:39Z 2024-02-02T09:53:56Z 3797
R2 2024-02-02T09:53:56Z 2024-02-02T10:00:42Z 406
SN 2024-02-02T10:00:43Z 2024-02-03T09:41:25Z 85242
R1 2024-02-03T14:41:18Z 2024-02-03T15:21:21Z 2403
R2 2024-02-03T15:21:21Z 2024-02-03T15:25:31Z 250
SN 2024-02-03T15:25:33Z 2024-02-04T15:36:25Z 87052
R1 2024-02-04T21:03:23Z 2024-02-04T22:07:34Z 3851
R2 2024-02-04T22:07:34Z 2024-02-04T22:13:33Z 359
SN 2024-02-04T22:13:34Z 2024-02-06T10:01:37Z 128883
R1 2024-02-06T15:05:12Z 2024-02-06T16:20:33Z 4521
R2 2024-02-06T16:20:33Z 2024-02-06T16:33:50Z 797
SN 2024-02-06T16:33:51Z 2024-02-08T07:54:13Z 141622
R1 2024-02-08T08:35:53Z 2024-02-08T09:23:37Z 2864
R2 2024-02-08T09:23:37Z 2024-02-08T09:30:21Z 404
SN 2024-02-08T09:30:22Z 2024-02-09T13:57:48Z 102446
R1 2024-02-09T14:33:36Z 2024-02-09T15:16:17Z 2561
R2 2024-02-09T15:16:17Z 2024-02-09T15:22:17Z 360
--8<---------------cut here---------------end--------------->8---
Attachment:
grok-logs-ftpsync
Description: Binary data