Upgrade to Chess.com Premium!

Using Rybka for Top 3 match analysis with ChessAnalyse

Some issues have been reported (http://www.chess.com/groups/forumview/fritz-xi-top-3-matchup-benchmark-stats , #133) in connection with the use of Rybka for constant time/ply analysis with ChessAnalyse, which have to be investigated further and might result in a change of ChessAnalyse's engine control.

To investigate the behavior of Rybka in Top 3 match analysis, I will perform some tests, analysing the engine's output and compare it to the uci specification and another engine (Houdini).
Let's do a manual (Command prompt) top 3 match analyis of the following (book) position:

 

 

 

 

 

 

 

 

 

 

 

 

 

1.) Starting Deep Rybka 4 w32 in a command shell (cmd.exe) and entering the following uci commands:
ucinewgame
// indicates, that a new game has to be analysed)
setoption name MultiPV value 3
// this sets MultiPV mode to 3)
position fen rn1q1rk1/pbp1bppp/1p1ppn2/3P4/2P5/P1N1P1N1/1P2BPPP/R1BQK2R b KQ - 1 9
// defines the starting position for analysis as a FEN string
go movetime 30000
// analyse exactly 30 seconds; this is the same command as launched by ChessAnalyse, when the parameters Time sec = 30 and Depth plies = 0.

Rybkas output is as follows:
... (first few hundred lines have been omitted) ...
info currmove b7c8 currmovenumber 27
info currmove f6e8 currmovenumber 28
info currmove h7h5 currmovenumber 29
info depth 12 time 7380 nodes 546967 nps 74114
info depth 13
info currmove c7c6 currmovenumber 1
info time 8112 nodes 604803
info time 9126 nodes 682393
info time 10140 nodes 755009
info multipv 1 depth 13 score cp -25 time 10344 nodes 770950 nps 74531 pv c7c6 e3e4 b8d7 e1g1 c6d5 c4d5 a8c8 c1e3 f8e8 d5e6 f7e6 a1c1 d7e5 d1a4 f6g4 a4a7 g4e3
info multipv 2 depth 12 score cp -17 time 4712 nodes 324823 nps 68935 pv b8d7 e1g1 c7c6 d5e6 f7e6 f2f4 d6d5 d1c2 a7a5 c1d2 a8c8 b2b3 d8c7 g3h5 f6h5 e2h5 d7c5
info multipv 3 depth 12 score cp -23 time 5040 nodes 349838 nps 69412 pv e6d5 c4d5 c7c6 e3e4 c6d5 e4d5 b8d7 e1g1 a8c8 c1e3 g7g6 a1c1 d7e5 d1d4 c8c3 c1c3 f6d5 c3c1 d5e3 d4e3
info currmove b8d7 currmovenumber 2
info time 11154 nodes 821657
info time 12168 nodes 896263
info multipv 1 depth 13 score cp -25 time 10344 nodes 770950 nps 74531 pv c7c6 e3e4 b8d7 e1g1 c6d5 c4d5 a8c8 c1e3 f8e8 d5e6 f7e6 a1c1 d7e5 d1a4 f6g4 a4a7 g4e3
info multipv 2 depth 13 score cp -25 time 12809 nodes 937644 nps 73201 pv b8d7 e1g1 c7c6 e3e4 c6d5 c4d5 a8c8 c1e3 f8e8 d5e6 f7e6 a1c1 d7e5 d1a4 f6g4 e3d4
info multipv 3 depth 12 score cp -23 time 5040 nodes 349838 nps 69412 pv e6d5 c4d5 c7c6 e3e4 c6d5 e4d5 b8d7 e1g1 a8c8 c1e3 g7g6 a1c1 d7e5 d1d4 c8c3 c1c3 f6d5 c3c1 d5e3 d4e3
info currmove e6d5 currmovenumber 3
info time 13182 nodes 965895
info multipv 1 depth 13 score cp -25 time 10344 nodes 770950 nps 74531 pv c7c6 e3e4 b8d7 e1g1 c6d5 c4d5 a8c8 c1e3 f8e8 d5e6 f7e6 a1c1 d7e5 d1a4 f6g4 a4a7 g4e3
info multipv 2 depth 13 score cp -25 time 12809 nodes 937644 nps 73201 pv b8d7 e1g1 c7c6 e3e4 c6d5 c4d5 a8c8 c1e3 f8e8 d5e6 f7e6 a1c1 d7e5 d1a4 f6g4 e3d4
info multipv 3 depth 13 score cp -27 time 13542 nodes 995587 nps 73518 pv e6d5 c4d5 c7c6 e3e4 c6d5 e4d5 b8d7 e1g1 g7g6 c1e3 a8c8 a1c1 d7e5 d1d4 c8c3 b2c3
info currmove h7h6 currmovenumber 4
info time 14196 nodes 1036522
info time 15210 nodes 1103169
info currmove a7a5 currmovenumber 5
info time 16224 nodes 1169817
info currmove b8a6 currmovenumber 6
info time 17238 nodes 1236465
info currmove f8e8 currmovenumber 7
info currmove c7c5 currmovenumber 8
info currmove d8d7 currmovenumber 9
info currmove f6d7 currmovenumber 10
info currmove b8c6 currmovenumber 11
info currmove f6d5 currmovenumber 12
info currmove b7d5 currmovenumber 13
info currmove f6e4 currmovenumber 14
info currmove f6g4 currmovenumber 15
info currmove b7c6 currmovenumber 16
info currmove b7a6 currmovenumber 17
info currmove e6e5 currmovenumber 18
info currmove g8h8 currmovenumber 19
info time 18252 nodes 1299134
info currmove b6b5 currmovenumber 20
info time 19266 nodes 1398608
info time 20280 nodes 1471224
info time 21294 nodes 1549809
info time 22308 nodes 1624415
info time 23322 nodes 1720905
info time 24336 nodes 1805458
info time 25350 nodes 1892995
info time 26364 nodes 1966606
info time 27378 nodes 2043201
info currmove a7a6 currmovenumber 21
info currmove g7g6 currmovenumber 22
info currmove f6h5 currmovenumber 23
info currmove g7g5 currmovenumber 24
info currmove d8c8 currmovenumber 25
info currmove d8e8 currmovenumber 26
info currmove b7c8 currmovenumber 27
info currmove f6e8 currmovenumber 28
info currmove h7h5 currmovenumber 29
info depth 13 time 28222 nodes 2095655 nps 74256
info depth 14
info currmove c7c6 currmovenumber 1
info time 28392 nodes 2108854
info time 29406 nodes 2195397
info multipv 1 depth 13 score cp -25 time 29984 nodes 2241155 nps 74745 pv c7c6
e3e4 b8d7 e1g1 c6d5 c4d5 a8c8 c1e3 f8e8 d5e6 f7e6 a1c1 d7e5 d1a4 f6g4 a4a7 g4e3
bestmove c7c6 ponder e3e4

Obviously the engine works in the following way:
Every possible move (29 in the position above) is analysed to a certain ply depth.
The analysis begins with the line
info depth [newDepthLevel]
and ends with the line
info depth [newDepthLevel] time ... nodes ... nps ...
Within such a depth level section there are one or more blocks displaying the information for the best 3 moves. They start with
info multipv [k] depth ... score ... cp time ... nodes ... nps ... pv ...
where [k] is 1, 2 and 3.
As one can see from the last lines of the output, the engine has just switched to start analysing depth level 14 for the first move, when the 30 sec were spent.
But, instead of giving all 3 multipv informations, as claimed in the uci specification (http://download.shredderchess.com/div/uci.zip where it reads:
"...in k-best mode always send all k variants in k strings together..."), only the information for multipv 1 is shown, with slightly different (higher) values for time, nodes and may be even different score cp, which may lead to the effects described in the link in the beginning, since ChessAnalyse always takes the last output for each multipv line.

The same effect occurs in an analysis to a certain ply depth. The corresponding uci command would be:
go depth 11
for example. Trying this, I noticed that an analysis to depth 13 for the above position last a significant time longer than the 30 sec analysis, which has already reached depth level 13, but it seems to be that all 29 moves are analysed to depth 13 with this latter command, while with the movetime command only the best 3 moves are analysed to this level.

There is still another possibility to do a 30 sec analysis with an uci engine. By typing the following command
go infinite
an infinite search is launched, which has to be interrupted with the command
stop
Unfortunately Rybka seems to not responding to the stop command in any mode...
(may be anybody could try this also and post something here..)

I did the same commands with the "Houdini 1.03a w32" engine and found everything fine.
Note, that the ending line of a ply depth section:
info depth [newDepthLevel] time ... nodes ... nps ...
is missing in Houdini's output, but the 3 multipv infos are always shown together and the engine reacts to the stop command immediately.

Comments


  • 21 months ago

    LegoPirateSenior

    Steve wrote about Rybkas: "They don't seem to cycle through lines fast enough to reach a decent depth."

    It has been speculated that this is a result of Rybka spending more time on position evaluation that implicitly might do additional searches (effectively adding a couple of plies to the reported depth). Some musings on that subject by a fellow who run an entire farm of Rybkas during matches against CC champions on chessgames.com (computers explicitly allowed) can be found here:

    http://mysite.verizon.net/vzesz4a6/current/id7.html

    http://mysite.verizon.net/vzesz4a6/current/id18.html

    Both somewhat dated, but interesting.

  • 21 months ago

    gwhuebner

    Steve wrote: "...In a given middlegame position, after 30 seconds Rybka may reach depth = 17 whereas a faster engine such as Houdini for instance might well get to depth = 18, or even 19."

    Well, I think the analysis depth announced by the engine is in some respects eyewash, because in the given time the engine (no engine) can calculate all possible variations to depth 18 or 19. So, simply omitting more branches will lead to higher final depths. That says nothing about the strength of the engine. The art of analysing is rather omitting the bad branches... - even the number of analysed positions (which an engine normally announces, too, is not too meaningful, because the art of analysing is not so much calculate the value of a position quickly but rather as exactly as possible...

    To get final clarity, one possibility might be to compare engines directly in a 30 sec/move tournament...

    2010-08-28: 6 games mini tournament Rybka vs. Houdini accomplished: (Result 4:2) http://blog.chess.com/view/small-engine-tournament-rybka-vs-houdini

  • 21 months ago

    SteveCollyer

    I have Rybka 3 & Deep Rybka 4 & both are pretty useless at 30 second top 3 match up, due to some problem with the search algorithm.  This is why I simply don't ever use them.

    They don't seem to cycle through lines fast enough to reach a decent depth.  It's painful to watch in comparison to many other engines on the same system.

    In a given middlegame position, after 30 seconds Rybka may reach depth = 17 whereas a faster engine such as Houdini for instance might well get to depth = 18, or even 19.

  • 21 months ago

    gwhuebner

    Rybka's MultiPV bug has now been fixed within ChessAnalyse (in principle the way, LegoPirateSenior suggested: only the last complete block of MultiPV lines is used within ChessAnalyse) and the sample analysis of Estrin's ten 20+ games of WCCC 1972 has been done again. (Download has been updated)
    http://www.chess.com/download/view/wccc-1972-estrin-analysed-with-chessanalyse-26

  • 22 months ago

    LegoPirateSenior

    Based on your description, indeed, buffering has nothing to do with the non-responsiveness to 'stop' (I believe that the the command terminal I/O is line buffered). I'll take a look at Rybka 3 next time I run Windows.

    As to the other problem, the missing move in multi-PV mode can be worked around if you keep the previous results until they are obsoleted. Consider this portion of the log:

    info multipv 1 depth 13 score cp -25 time 10344 nodes 770950 nps 74531 pv c7c6 e3e4 b8d7 e1g1 c6d5 c4d5 a8c8 c1e3 f8e8 d5e6 f7e6 a1c1 d7e5 d1a4 f6g4 a4a7 g4e3
    info multipv 2 depth 12 score cp -17 time 4712 nodes 324823 nps 68935 pv b8d7 e1g1 c7c6 d5e6 f7e6 f2f4 d6d5 d1c2 a7a5 c1d2 a8c8 b2b3 d8c7 g3h5 f6h5 e2h5 d7c5
    info multipv 3 depth 12 score cp -23 time 5040 nodes 349838 nps 69412 pv e6d5 c4d5 c7c6 e3e4 c6d5 e4d5 b8d7 e1g1 a8c8 c1e3 g7g6 a1c1 d7e5 d1d4 c8c3 c1c3 f6d5 c3c1 d5e3 d4e3
    ...
    info multipv 1 depth 13 score cp -25 time 10344 nodes 770950 nps 74531 pv c7c6 e3e4 b8d7 e1g1 c6d5 c4d5 a8c8 c1e3 f8e8 d5e6 f7e6 a1c1 d7e5 d1a4 f6g4 a4a7 g4e3
    info multipv 2 depth 13 score cp -25 time 12809 nodes 937644 nps 73201 pv b8d7 e1g1 c7c6 e3e4 c6d5 c4d5 a8c8 c1e3 f8e8 d5e6 f7e6 a1c1 d7e5 d1a4 f6g4 e3d4
    info multipv 3 depth 12 score cp -23 time 5040 nodes 349838 nps 69412 pv e6d5 c4d5 c7c6 e3e4 c6d5 e4d5 b8d7 e1g1 a8c8 c1e3 g7g6 a1c1 d7e5 d1d4 c8c3 c1c3 f6d5 c3c1 d5e3 d4e3

    The last line in the the above reports repeats (so does the first). If Rybka conformed to the specifications, the last report would've read:

    info multipv 1 depth 13 score cp -25 time 29984 nodes 2241155 nps 74745 pv c7c6
    e3e4 b8d7 e1g1 c6d5 c4d5 a8c8 c1e3 f8e8 d5e6 f7e6 a1c1 d7e5 d1a4 f6g4 a4a7 g4e3
    info multipv 2 depth 13 score cp -25 time 12809 nodes 937644 nps 73201 pv b8d7 e1g1 c7c6 e3e4 c6d5 c4d5 a8c8 c1e3 f8e8 d5e6 f7e6 a1c1 d7e5 d1a4 f6g4 e3d4
    info multipv 3 depth 13 score cp -27 time 13542 nodes 995587 nps 73518 pv e6d5 c4d5 c7c6 e3e4 c6d5 e4d5 b8d7 e1g1 g7g6 c1e3 a8c8 a1c1 d7e5 d1d4 c8c3 b2c3
    bestmove c7c6 ponder e3e4


  • 22 months ago

    gwhuebner

    I've now decided to work only with the top move scores of the engine, when a complete cycle of multipv lines (1, 2, 3) has been written by the engine in order to circumvent the multipv bug of Rybka 4. 

    With this change, the Rybka bugs, mentioned above should lead to no issues in connection with ChessAnalyse, anymore.

    Furthermore I will implement in version 2.6, yet, the visualisation of the engine's calculation process by continuously updating the move score region of ChessAnalyse with the current results of the multipv cycles. (This may be common practise in other GUIs, too).

  • 22 months ago

    gwhuebner

    LegoPirateSenior, you can even reproduce the "stop-bug" with the free (demo) version Rybka 2.3.2a mp 32 bit, downloadable here:
    http://www.rybkachess.com/index.php?auswahl=Demo+version
    (not the "multipv-bug" - this one seems to be new in Rybka 4, and may be Rybka 3, too, but I haven't got that version).

    Have you tried the above with your Rybka 3? From a logical point of view the stop-bug should be present in Rybka 3, too. What were your results?
    (By the way, the engine can be stopped in a forced way by pressing Ctrl+C within the command prompt).

    Well, I think it could not be something with my buffering or my system. I simply start Rybka 4 in a DOS command shell (cmd.exe), type "go infinite" (followed by the Enter button), wait 1 or 2 seconds and then type "stop" (+ Enter) and the engine won't stop, while any other uci engine will stop immediately (I tried Houdini, Fire, Shredder 12, and others). It's definitely a bug in the engine. Or, some parameter may have to be adjusted (very unlikely). It may also be a bug only in my special version (build) of Rybka 4 (bought Deep Rybka 4 without Aquarium GUI).

    In the meantime I'm trying to mix up the Rybka forum a little bit ;-)
    http://rybkaforum.net/cgi-bin/rybkaforum/topic_show.pl?tid=17231 
    http://rybkaforum.net/cgi-bin/rybkaforum/topic_show.pl?tid=18555 

    But the philosophy of the Rybka selling connection seems to be:
    No updates or hotfixes. Let the customer buy the next version in the hope, that bugs will be fixed there...

  • 22 months ago

    LegoPirateSenior

    "Unfortunately Rybka seems to not responding to the stop command in any mode... (may be anybody could try this also and post something here)"

    Unfortunately, I have only Rybka 3.

    Even though in my real life I wrote lots of code for interprocess communications, I still get occasionally bitten by buffering issues. Hence a quick suggestion: perhaps you might want to check what kind of buffering you have on your side of the I/O channel used to talk to the engine. It may need to be set to line buffering mode or to no-buffering.

    Apologies if you already have done this...

Back to Top

Post your reply: