Library regression tests

[jjgarcia]

[Cc from comp.lang.lisp]

I know that coding for free is quite an achievement, and that it is daring to ask free developers for something, but I thought it would be nice for once to be on the asking side

I am setting up an automated process for downloading, building and testing relevant, popular, or simply LARGE software made in Common Lisp. The goal is to ensure that the code is not broken by changes in ECL. It is a selfish goal, but probably exportable to other implementations.

http://ecls.sourceforge.net/logs_lib.html

What I would like to ask CL developers is to begin shipping regression tests with their libraries, and perhaps agree on an standard so that one can programatically check whether a library builds or not. This feature could be even included in ASDF, asdf-install, etc, just like "make check" is now a de-facto standad in Autoconf.

The agreement can be very simple: a package name (cffi-test for cffi, etc) and a function that is to be executed and returns the number of tests run, number of failures, and perhaps a descriptive list of the test to be pretty printed.

Do you think it makes sense?

Juanjo

[ramarren]

What I would like to ask CL developers is to begin shipping regression tests with their libraries, and perhaps agree on an standard so that one can programatically check whether a library builds or not. This feature could be even included in ASDF, asdf-install, etc, just like "make check" is now a de-facto standad in Autoconf.

The agreement can be very simple: a package name (cffi-test for cffi, etc) and a function that is to be executed and returns the number of tests run, number of failures, and perhaps a descriptive list of the test to be pretty printed.

There already is such a feature included in ASDF, namely the test-op operation. I see about seventy systems from clbuild implementing them, which is about a fifth of them. It should probably be more, but it is a start. And that includes in particular cffi, so I am not sure what your particular example means.

[gugamilare]

I believe standardization is always good if it is well planned, because it avoids needing to change someone else's code.

The package naming is a not a bad idea, but using the 'asdf:test-op is a very reasonable agreement and more flexible since it is a generic function. Since there are already people that like to return T on success and NIL on failure, we could agree on returning four values from each call to this generic function, only one required (values all-tests-passed-p &optional number-of-tests-run number-of-failures test-descriptive-list) (Actually, I didn't quite get the idea of the last returning value). And one more agreement: print all information to *standard-output* (this is already said in ASDF's manual).

This requirement would normally take about 5 lines in the .asd system definition, since almost all (even trivial) libraries already have a "test-this-library" function. We could send these suggestions and patches to some libraries.

[dmitry_vk]

I didn't actually know about asdf:test-op. I think that automated testing of libraries makes perfect sense (ideally, it should not be limited to ECL but should be done with other implementations also).
It would also make sense to have a way to notify developers that the tests have failed for their libraries (probably via bugzilla/launchpad-like web-site).

[jjgarcia]

using the 'asdf:test-op is a very reasonable agreement and more flexible since it is a generic function... we could agree on returning four values from each call to this generic function,

I did not know about test-op. It is great to have already something there as a starting point and as you say, it is only a matter of enforcing that this function returns something meaningful.

Actually, I didn't quite get the idea of the last returning value.

Well one possiblity is to write everything to *standard-output*, but I was thinking more on a kind of list of tests that could also be programmatically manipulated.

[jjgarcia]

Using the 'asdf:test-op is a very reasonable agreement and more flexible since it is a generic function. Since there are already people that like to return T on success and NIL on failure...

Actually, I have found the opposite. The return value of ASDF:OOS on ASDF:TEST-OP is, currently, quite meaningless, because each programmer has their own convention. See real-life examples below:

Code: Select all

> (asdf:oos 'asdf:test-op 'alexandria)
[...]
No tests failed.
NIL
> (asdf:oos 'asdf:test-op 'flexi-streams)
[...]
Some tests failed.
NIL
>

[gugamilare]

Using the 'asdf:test-op is a very reasonable agreement and more flexible since it is a generic function. Since there are already people that like to return T on success and NIL on failure...

Actually, I have found the opposite. The return value of ASDF:OOS on ASDF:TEST-OP is, currently, quite meaningless, because each programmer has their own convention. See real-life examples below:

Code: Select all

> (asdf:oos 'asdf:test-op 'alexandria)
[...]
No tests failed.
NIL
> (asdf:oos 'asdf:test-op 'flexi-streams)
[...]
Some tests failed.
NIL
>

One thought. Most libraries that implement tests use some test framework available (lift, rt, fiveam, ...). If we create functions able to extract information from each test framework, then each library would have almost work to return something valuable from the asdf:test-op method.

For instance, this implements a test result reporter for lift (the format of the "descriptive list" can be changed):

Code: Select all

(defun lift-report-result (&optional (test-result lift:*test-result*))
  (flet ((get-info-from-report (report)
           `(:testsuite ,(type-of (lift::testsuite report))
                        :test-name ,(lift::test-method report)
                        :condition ,(lift::test-condition report)
                        :code ,(lift::test-report-code (lift::testsuite report) (lift::test-method report)))))
    (with-accessors ((expected-failures lift:expected-failures)
                     (expected-errors lift:expected-errors)
                     (failures lift:failures)
                     (errors lift:errors)
                     (tests-run lift::tests-run))
        test-result
      (values (not (or errors failures))
              (length tests-run)
              (+ (length errors) (length failures))
              `((:tests-run ,@(mapcar #'(lambda (test)
                                          `(:testsuite ,(first test)
                                                       :test-name ,(second test)))
                                      tests-run))
                (:errors ,@(mapcar #'get-info-from-report errors))
                (:failures ,@(mapcar #'get-info-from-report failures))
                (:expected-errors ,@(mapcar #'get-info-from-report expected-errors))
                (:expected-failures ,@(mapcar #'get-info-from-report expected-failures)))))))

To test an example of output, just (asdf:oos 'asdf:test-op :lift) and (lift-report-result) (its too long to be posted here).