I am using obspy-sds-report script to generate html report of my SDS archive. But I would like to limit the depth of the scan to the last month or current year to avoid very long process and "no data within xx days" in the HTML figure for stations that are not any more used/available.
The options that I am using are : *--check-backwards-days 15 --check-quality-days 15*, but the scan seems to check all the SDS archive...
Is there a way to force the scan to a limited time span ?
"--check-backwards-days" should be the option you want. How does it not
work as expected? The first discovery of ids will still find stations
that have no data in the given search range though. You can control the
lookup somewhat with "--location" and "--channel", if that isn't enough,
you could set up a dummy latency file and always use it together with
"--update" option to only look at a fixed set of channels. Check the
output txt file of one of your runs, it is comma separated:
network, station, location, channel, latency, percentage, gap count
You should be able to set those up with dummy numbers.
Hello, Thanks for your answers. I already tested the update option, but it only works for latencies nor for the gaps... And when I run it without this option the script scans all my SDS (containing data since 2013)...the backward option does not seems to be considered in the scan, only in the results.
if my previous response wasn't sufficiently clear: in the "full"
operation mode (without using "--update"), the SDS filesystem might be
traversed fully (at least the folder structure) to identify what NSLC
(channel SEED Ids) are in the SDS overall and then it will check each
channel of the whole SDS for files according to the "backwards days"
option (and show them as grey if no data in that span).
Subsequent "update" runs, will not search for NSLC again, but rather
look at the saved output txt file and then check the NSLC listed there.
So if you want to only scan specific NSLC / SEED IDs, set up such an
output file fram scratch or modify the one you already got and then ever
only use the "update" option on that initial output file.
No idea what is is about gaps not being checked, they should be checked
and reported in both the "full" and "update" mode i believe.
There is certainly room for improvement, discussion about enhancements
we should take to github, I would propose.
The way I use it is do a "full" report only every now and then
(depending on how fast the filesystem is, how big the SDS archive etc),
e.g. once per day, so that new stations get eventually picked up.
Otherwise do "update" much more frequently, e.g. every 5 mins, depending
on how up-to-date it has to be (e.g. when reenabling a station in the
field and checking if data come back in).
If you do it like that (e.g. with cronjobs) you need to make sure that a
new full run output (which is much longer running) is not overwritten
again by an update run that was based on an old output file (I hope it's
clear what I mean). If I remember correctly, I kill all running "update"
processes immediately after completing a "full" run in my cronjob.
I have tried to configure the file output from the "full" run to only use it with an update mode after.
But the update mode does not check the gaps, only the latencies...not the way I want to use it because I do have some scripts that patch the gaps automatically.
The best way should be to have the update mode fully working (gaps & latencies) with file, or as Fred mentionned any solution to give for exemple the Year of the SDS to be computed, OR to limit the full scan to the backward days given in the option. It seems that there is in the script a way to process the time information
`
but maybe this is not fully implemented from the obspy-sds-report…`
This tool is really nice, but it could be even better if we would add a "back" option in the scan...
ah I see now. indeed gaps are not updated in the fast mode (mostly to
save time). It should be trivial to add an option "--update-gaps" to
also update gaps when in the "update" mode.
I think we should continue this on github now as it's development related.