How does a test failure work?
I know PASSFAIL can be used to set the final status of 0x0 (fail) or 0x1 (pass)... and I know the test should start with a PASSFAIL 0x0
The chip tests you coded use IF statements to set PASSFAIL and perform a SHOSTR that displays a custom fail message. Then a moment later the screen clears and becomes the common red failure screen.
But according to the code, after that IF, the test continues.... so I think you have some non-obvious "code short circuiting" check points going on that looks for PASSFAIL failure state and chooses not to continue the test if the test has already failed? Hmmmm... though it feels like a bad coding practice... could PASSFAIL be set to 0x1, then set back to 0x0 without the test ending?
Q1: When is the PASSFAIL checked and test stopped?
Q2: Can my test leave ever the custom progress/pass/fail details on the screen or will green/red screen always replace my script's output?
Passfail is simply a way to tell the tester to switch to either the pass screen or the fail screen when the test ends. It is just a stored value of pass or fail. It gets set like any other operation, that is, each opcode is performed in order. You never need to set pass fail if you don't want to. It will autofail however if you don't set pass. In the tests I have made, I set it to pass automatically at the start. If something doesn't pass I set fail and leave it because even if something else passes later that part still failed. You aren't required to fail a test if you don't want to. Say you are testing two different things but you don't care if one of them fails, well, you can just have it show text that it failed and still keep it as passing if you want to. It's basically just a variable that will always fail unless you set it to pass.
Oh, I noticed this in the scripting docs page:
```
ENDTEST 0x1 0x0 0x0
```
Is that a real standard or an old standard?
We've been talking about using PASSFAIL instead of using ENDTEST's first argument.
Ultimately my question is about best practice for failing a test script.
I was working with your 4164 and 74LS08 scripts, and they handle errors differently.
That lead to my confusion about the failure pattern, the language itself, and I mistakenly started to wonder if there was some extra magic going on.
It makes sense that there is nothing special about the PASSFAIL... now that I see the differences in the two scripts.
For posterity, I'll point out the two approaches that you've coded.
Pattern "One ENDTEST" (checks show good/bad, set failure & continue, one ENDTEST at bottom)
The 74LS08.chip script checks all truth table variations for each of 4 logic gates, and displays "good" or "bad" strings for each gate... but never ends the test until the last line of the script. If we just wanted to know chip good/bad, the first failure could immediately ended the test.
```
[...]
PASSFAIL 0x1 0x0 0x0
SHOSTR 0x0 0x0 0x28
UDSCRN 0x0 0x0 0x0
DELMS 0x3E8 0x0 0x0
SHOSTR 0x1 0x0 0x3C
RDPIN 0x3 0x1 0x0
IF 0x3 #R1 0x0
PASSFAIL 0x0 0x0 0x0
SHOSTR 0x2 0x0 0x3C
ENDIF 0x0 0x0 0x0
SETPIN 0x1 0x1 0x0
RDPIN 0x3 0x1 0x0
IF 0x3 #R1 0x0
PASSFAIL 0x0 0x0 0x0
SHOSTR 0x2 0x0 0x3C
ENDIF 0x0 0x0 0x0
[...]
ENDTEST
```
Pattern "Multiple ENDTEST" (multiple checks of IF..PASSFAIL..ENDTEST..ENDIF)
The 4164.chip script has 2 loops and an inner IF condition that immediately ends the script on the first failure. If it were coded like the 74LS08.chip, it might count all the failures, NOT do an ENDTEST in the IF condition, showing counts and the last line would be ENDTEST.
```
PASSFAIL 0x1 0x0 0x0
[...]
RDPIN 0xE 0x2 0x0
SETPIN 0xF 0x1 0x0
SETPIN 0x4 0x1 0x0
IF 0x1 #R2 0x1
PASSFAIL 0x0 0x0 0x0
ENDTEST 0x0 0x0 0x0
ENDIF 0x0 0x0 0x0
[...]
ENDTEST
```
I like the "Multiple ENDTEST" pattern best.
... because I only care if the chip is good or bad (decide as quick as possible because I could be testing a pile of chips)
... plus managing/queuing the good & bad strings makes the script a little harder to read.
Yeah, your preferred method is what I would use if I had a lot of them to test but sometimes I just like to see what's going on. I did various patterns for some of the tests so that people could get a good idea of some different ways to set them up. A lot of it is preference but it could be useful too if there are different kinds of tests someone wants to run. Like some of the RAM tests are full all address tests which take a while. Some of them are very short only testing like only testing for unwanted bit switching in rows or columns. Those I did give different names to though for the most part. You're really going to like the functions. You only need to declare functions after ENDTEST and then anywhere above end test you just use CALL (name). No other arguments needed for functions.. Just FUNC name, ENDFUNC, CALL name. that's it. I have the OSX version done and just need to port it to Windows now. It should take me about a day or two.
@nivbot The functions idea sounds very helpful !!
Q: Are registers the mechanism for passing values into a FUNC?
Q: A register can be used to return a value from a FUNC?
The industry word "function" implies that it returns a value, and normally a "procedure" does not.
The word "function" is a fine choice, especially if I can do the normal things like pass value(s) in and receive value(s) back out.
Oh, and I can still do your ENDTEST trick inside a function? ...and the test ends immediately?
@aadhoc4370 Oh yeah, everything works the same. the only difference is you can now make functions. The only reason to make it after ENDTEST was to have a paper place for writing functions. so it isn't in the middle of the code. If you want to know how the function names work it's fairly simple. I just replaced the required arguments with ASCII Bytes. So you now have a maximum of 12 characters for your function name. In the GUI it uses its name, in the actual code on Vetus, the gui translates the name bytes located in argument 1 of the name into a line number. So that way Vetus has an easy time jumping to the code and we get to see pretty words.
Oh, there is no special return value from a func but there's no reason you can't use a register to output anything you want to From the function. Maybe in the future we can look into adding parameters or return values.
@aadhoc4370 Ooooh inside of a function? You know, I didn't account for that but I think as long as you have your functions after the first ENDTEST it should work as normal.
@nivbot I like the opportunity to put the ENDTEST inside functions, but the tradeoff is going to be the main logic (and its readability) must expect that the functions may not return.
Imagine if the function decides to end the test, the main script's "high level" logic flow gets thwarted. If done in a readable and standard fashion, this could be useful and become a standard the community endorses. In other words, maybe we generally decide that an ENDTEST can be done in the "main test" logic and all "sub-test" functions. Sure that makes sense, but only because we have accepted that "tests" could fail at any moment... thus any part of the code we consider a test could "just stop". That's probably a good and easy standard.
Notes/caveats:
1) Anywhere ENDTEST occurs, the PASSFAIL must have already been set too. Imagine a test function running, encountering a problem, and doing ENDTEST. If PASSFAIL wasn't set, the function probably just declared that "the main test" just succeeded. Obviously it should be a "PASSFAIL 0x0 0x0 0x0; ENDTEST 0x0 0x0 0x0" (Set PASSFAIL to 0 for failure).
2) If the main test logic calls any custom function that isn't "testing" anything, then the function is not a "sub-test", so that function should NOT use PASSFAIL or ENDTEST. Imagine having high level script call `testAndGATE`, then `displayResult`, then `testOrGATE`, ... just to be frustrated when `testOrGATE` never gets called. Well obviously `testAndGATE` failed right? Not necessarily. Maybe `displayResult` used ENDTEST?!? Silly example, but the point is when problems/debugging occur, we need standards that make the code more obvious and maintainable. In higher level languages we use words like `Verify`, `Confirm`, `Ensure` and `Assert` to convey that the method will only return if successful (and won't if not). In this context "Test" is a great word to mean "only return if successful"... assuming the "fail-on-first-failure" pattern is used.
@aadhoc4370 The main function doesn't really need to care about whether the test ends right away or in some function or whatever. Once the test ends all of the pins are set to Hi-Z and the entire test ends a new screen opens. The end test screen has very little code and only looks for the test result and shows it. So, no matter where you end the test everything is shut down. It either passed or failed. Now, if you want to do other things like showing results before ending the test that might be something else entirely but it seems like it still wouldn't make a difference.
PS, I plan to make the the END TEST screen editable I just haven't yet. I wasn't planning to but I recently decided it would be nice.
@nivbot I'm not sure you should change the END TEST screen.
Red/Green is fine with text as you have it.
Seriously, I would leave it alone.
Those are standard screens we can get used to.
Now, if you want to allow the test to provide a string to put at the bottom of the success/fail screen, that doesn't take away from the standard very much, but what could possibly be worth showing the user about the success or failure that they would find useful? I assume we're all thinking "Green" means keep the chip, "Red" means throw it away immediately. Whether gate 3 failed or gate 4 failed... I suspect the users will accept that any fail means they shouldn't be using the chip. I can't imagine someone saying "but I still want to use the working gates", or the 6502 instructions that DO work.
IDEA: If a test REALLY wants to display some detail about the failed test, maybe let them display it on the screen and ENDTEST, and after a ~5 second delay, flip to the Green/Red screen. BUT, put on the screen a "right button for details" message, so if they push the right button... it goes back to the test's end display & messages. That way we get the standard END TEST screen, and if it was a long test and we missed the test messages, we can switch to it?
Flexibility for we programmers always sounds good, but the reality of use-case scenarios should override breaking those good standards.
Maybe you're thinking of something I haven't considered?
@aadhoc4370 I wouldn't throw the chip away immediately. Tests aren't foolproof. Even the best scientific tests have an error threshold. I'm not intelligent enough know how to figure out that threshold but it's there. Now it WOULD mean that you should test it a few more times and it is more likely than not bad.
@nivbot I'm pointing out that the success/fail result is what matters most.
Leave that END TEST screen alone since there will be hundreds of test scripts written by lots of people.
That's the vision right?
No matter who wrote the test, I think I/you need a standard success/fail screen.
Find another way to provide failure details (that's what I proposed above)