Remove stats file production

[David Davies (DJDavies2)]

Fix #291. It does a bit more than fix that issue in that it removes stats file generation altogether. This doesn't affect the ctests as they continue to pass but if there is some use for stats file generation from opsinputs that isn't covered by this then this change isn't going to be appropriate.

To be clear, the problem in #291 is that because the stats file is written to fort.7, running opsinputs in parallel can therefore result in concurrent writes to the same file although this is a transient failure.

Test output:

https://cylchub/services/cylc-review/taskjobs/david.davies/?suite=OpsinputsStatsFile

[Adam Maycock (adammaycock)]

I believe the "stats" file contains only a line informing how many obs were output to varobs. I can't see any use for this, so happy with this change.

[Michael Cooke (mikecooke77)]

I can see why these need to be removed. Is this functionality that was used in OPS? Could it have utility here?

I have made one comment about the location of the changes files but apart from that I'm fine with this change.

[Michael Cooke (mikecooke77)]

I think now this is modified it should move to the stubs directory as it has diverged from the original file copied from OPS.

[David Davies (DJDavies2)]

Do you mean just this file or the whole directory?

[Michael Cooke (mikecooke77)]

I think now this is modified it should move to the stubs directory as it has diverged from the original file copied from OPS.

[David Davies (DJDavies2)]

Do you mean just this file or the whole directory?

[Michael Cooke (mikecooke77)]

I think now this is modified it should move to the stubs directory as it has diverged from the original file copied from OPS.

[David Davies (DJDavies2)]

Do you mean just this file or the whole directory?

[David Davies (DJDavies2)]

I can see why these need to be removed. Is this functionality that was used in OPS? Could it have utility here?

I have made one comment about the location of the changes files but apart from that I'm fine with this change.

Stats files were certainly used by people, they are produced operationally and were used to check on things like obs counts and obs stats. Adam Maycock (@adammaycock) probably knows more about these uses than me but they are/were certainly used.

However if we are working with ODB's I would assume that in a Jedi context there isn't anything in a stats file that cannot be obtained from an ODB. This is probably true of OPS as well, but stats files predate ODB in OPS by many years.

The CODB metdb to odb tasks that generate odb's via OpsScr_CreateODB still generate stats files, although all they do is write out ob counts. However that code is not affected as that is still using OPS code.

[Adam Maycock (adammaycock)]

I can see why these need to be removed. Is this functionality that was used in OPS? Could it have utility here?
I have made one comment about the location of the changes files but apart from that I'm fine with this change.

Stats files were certainly used by people, they are produced operationally and were used to check on things like obs counts and obs stats. Adam Maycock (Adam Maycock (@adammaycock)) probably knows more about these uses than me but they are/were certainly used.

However if we are working with ODB's I would assume that in a Jedi context there isn't anything in a stats file that cannot be obtained from an ODB. This is probably true of OPS as well, but stats files predate ODB in OPS by many years.

The CODB metdb to odb tasks that generate odb's via OpsScr_CreateODB still generate stats files, although all they do is write out ob counts. However that code is not affected as that is still using OPS code.

The stats files generated by OPS were certainly useful and used for a number of monitoring applications. They contained mean / RMS o-b stats, and rejection percentages etc. The stats files currently generated by JOPA don't contain any of that, they only contain a "Total number of Varobs" line which is of limited use. So I don't see a problem with getting rid, as proposed here.

For info, there is an open discussion on getting OPS like stats from JOPA: MetOffice/jedi-discussion#71.