Evaluation tonica Turing Test
Evaluation tonica Turing Test
In 1950, famous computer scientist Alan Turing developed a test where a person was to find out whether a human being or only a computer simulation of a human being hides behind a system. If he or she is not able to do so, the system may be considered as intelligent. Initially this test was not more than a theoretical outline but was formulated more exactly later when artificial intelligence was considered a subdivision of computer science and became an independent academic field of work.
Our software tonica fugata learned the J. S. Bach harmonization style from his choral compositions with the aid of neural networks, a method of artificial intelligence. By applying the learned musical knowledge tonica creates new choral compositions for melodies it has never seen before.
The task in our Turing Test is to find out whether ten chorales were composed by Bach or by tonica fugata. The Turing Test can be taken online (no registration required) here:
The tonica Turing Test was performed anonymously during a period of two and a half weeks (Sep 21 to Oct 8, 2018) by capella-software customers presumably with musical (basic) knowledge.
In this period of time the test was performed 835 times. In 258 tests all ten chorales were evaluated, in all other tries participants canceled the test prematurely.
The average success rate was 62 %, e.g. the number of chorales correctly assigned to Bach or tonica fugata. A success rate of 100 % means that all chorales were correctly assigned and the test failed. A success rate of 50 % equals a „random guess“ of participants and thus a successful Turing Test.
It is interesting to compare the success rate of participants having completed the test with those of participants who canceled prematurely:
While the former correctly assigned 70 % of the chorales, the success rate of the latter was only 52 % which is only slightly above random guess. It can be presumed that participants having performed the entire test and thus shown larger interest in the test bring more previous knowledge.
Here are the results for the single chorales:
It can be seen that two Bach chorales (Nos. 3 and 6) were considered as computer compositions by the majority. All other chorales (amongst these all tonica chorales) were assigned correctly by the majority.
The tonica chorale which was most likely considered as Bach chorale was No. 2. It is a chorale following the tune “Herr, ich denk’ an jene Zeit”. Stated below the chorale composed by tonica fugata, followed by the Bach original:
In summary it can be stated that the tonica Turing test was (almost) successful for participants with less previous knowledge, e.g. participants were not able to tell “original” from “computer composition”. Looking at participants with more previous knowledge it can be seen that differences between original and computer generated compositions were still distinguishable by the majority. This means that the compositions showed characteristics helping persons with previous musical knowledge recognize them as original or computer compositions.
Take the opportunity to participate in the tonica Turing test and test your knowledge!