Welcome to Our Community

Some features disabled for guests. Register Today.

Catastrophic failure after update, problems probably not software

Discussion in 'Control Software' started by donald k wilson, Dec 10, 2022.

  1. donald k wilson

    Builder

    Joined:
    Nov 10, 2016
    Messages:
    49
    Likes Received:
    11
    I have encountered a "glitch" in the software during the running of cad files I will try to explain... This has happened 3 times since the latest update.

    The (open builds cam) software randomly drops or is dropped from communication from the usb port, in this case com3.

    The program stays running with the option to reconnect.
    First time reconnect was successful, starting the cut over file was successful and with a repeat of what was already run prior to the interrupt the file finished w/o additional issues.

    The second time, reconnect was successful but the file would not clear or reload. I had to close and restart openbuilds to clear the old file, reopen the file from the directory and then proceed as normal w/o any issues.

    The third time, reconnect was unsuccessful, I had to close open builds again, when restarted it opened with the previous file still on the console and the machine started to travel w/o prompting from command in a direction having nothing to do with the cut file. Stop job,abort and shutting the program off had no effect on the motion. I finally turned off the black box although by that time the machine had reached x limit, buried itself in the table and stopped moving y. I can't say if it stopped from black box disconnect or full stall from hitting limits and resistance. The movement pulled the wood blank out of the hold downs and carried it about 20" diagonally across the table where it eventually stopped.

    I have uploaded a photo of the table disaster and a portion of the file where the disconnect happened. I cannot begin to understand why this would happen, any thoughts or guidance would be greatly appreciated, I'm a bit hesitant to proceed until I can get a handle on the issue(s)

    Thank You...!

    IMG_1085.JPG
    IMG_1084.JPG
     
    #1 donald k wilson, Dec 10, 2022
    Last edited: Dec 10, 2022
  2. donald k wilson

    Builder

    Joined:
    Nov 10, 2016
    Messages:
    49
    Likes Received:
    11
    This just happened again while resetting the machine coordinates after the accident. There was no file loaded and no restart necessary just a dropped connection, port closed, detected a change in available ports com1, com3 and a reconnect on the top bar. no crazy machine movement after the reconnect.
     
  3. Gary Caruso

    Gary Caruso OpenBuilds Volunteer
    Staff Member Moderator Builder

    Joined:
    May 19, 2016
    Messages:
    1,193
    Likes Received:
    532
    Hi Donald, the best way to rule it out would be to download something like UGS Platform (any of the free grbl g-code senders) and run the G-code though that and see if the same things happens.
    I would think its more likely a hardware problem than a software one. USB cable, EMI, the port on the PC can all cause this issue.

    Regards
    Gary
     
    donald k wilson likes this.
  4. donald k wilson

    Builder

    Joined:
    Nov 10, 2016
    Messages:
    49
    Likes Received:
    11

    I was considering "all of the above" it seemed odd it would start happening with some regularity right after an update but that is a circumstantial association. I will begin the process of elimination. Thanks for your input!
     
  5. donald k wilson

    Builder

    Joined:
    Nov 10, 2016
    Messages:
    49
    Likes Received:
    11
    Another question... What might cause EMI and how far might the broadcast radius be? I don't have anything around but the computer, control boxes, dc converter and spindle and some fluorescent lights. No change in the environment with the exception of the software update. I have a spare usb cable I can swap and I can run the software out of my laptop as a check against the usb port in the computer. low hanging fruit...
    Thanks!
     
  6. Shell M. Shrader

    Builder

    Joined:
    Dec 9, 2022
    Messages:
    19
    Likes Received:
    13
    All kinds of things cause EMI. Our spindles and VFDs are the biggest culprits. You want the VFD's mains (as well as its power supply), and the actual spindle cable isolated from everything else as much as possible. Another source is the power supply for your controller... basically AC is noisy and many of the VFDs out there (huanyang for example) splatter signals across connected devices. I've seen a VFD decimate an RS485 signal for reference.

    When I read your message, the first thing I thought was, "gawd, I hate Windows". Any number of things cause disconnection issues there. I'm somewhat biased, but I would never, ever, run anything mission critical on a Windows machine.
     
    donald k wilson likes this.
  7. Gary Caruso

    Gary Caruso OpenBuilds Volunteer
    Staff Member Moderator Builder

    Joined:
    May 19, 2016
    Messages:
    1,193
    Likes Received:
    532
    I find grounding is often overlooked but good grounding of everything helps a bunch. And as Shell said get the spindle power away from the rest, maybe try running an air cut (run the program with the zero so high it never touches the surface) this can be done with no bit and spindle off and on to see if anything changes.
    USB cable is easy first move.
    Cheers
    Gary
     
  8. David the swarfer

    David the swarfer OpenBuilds Team
    Staff Member Moderator Builder Resident Builder

    Joined:
    Aug 6, 2013
    Messages:
    3,430
    Likes Received:
    1,907
  9. donald k wilson

    Builder

    Joined:
    Nov 10, 2016
    Messages:
    49
    Likes Received:
    11
    Thanks everyone, all possibilities and good tips. My vfd is about 3' away from the black box and the computer but the spindle power passes directly next to it in the wiring harness. The black box is mounted on the gantry to minimize wire travel, perhaps a change of location there.

    I run a humidifier in the shop but with forced hot air and wood products I am very dry. Static is another def consideration.

    I'll run a test file with the vac off and spindle pulled up, easy double check.

    My computer is constantly running windows updates, this has caused problems in other areas however this issue happened after an openbuilds update as well.

    I'm also suspicious of the usb card on the computer itself. The machine is old enough to be updated w/o much consideration anyhow.

    Fortunately the plate that was ruined was a crash test dummy so good luck there, it served its purpose. ;-) Fingers crossed, I'll update. Thank You!
     
    #9 donald k wilson, Dec 11, 2022
    Last edited: Dec 11, 2022
  10. donald k wilson

    Builder

    Joined:
    Nov 10, 2016
    Messages:
    49
    Likes Received:
    11
    Very informative, Thank You...!
     
    David the swarfer likes this.
  11. Peter Van Der Walt

    Peter Van Der Walt OpenBuilds Team
    Staff Member Moderator Builder Resident Builder

    Joined:
    Mar 1, 2017
    Messages:
    14,871
    Likes Received:
    4,283
    In CONTROL, click the Troubleshooting tab. Review the changelog. If you see that the Changelog for the update you installed mentions working on usb/serial - well then your symptoms may have been update related. But it doesn't does it.

    The last update was twofold: change to the included firmware binary that the flashing tool uses when someone runs the tool on a new X32 (updated firmware to address two X32 firmware bugs) [full details here Bug: Probe queue stuck · Issue #277 · OpenBuilds/OpenBuilds-CONTROL], and
    a minor change to preset machine profiles (only used during initial machine set up - remember the old step where you picked your machine from the list?) [documented in more details at Check $11 value in profiles · Issue #279 · OpenBuilds/OpenBuilds-CONTROL].

    Neither of which has any relevance to comms...

    Changelogs are shown openly. Mistakes can happen yes. Updates can have bugs. But look at the changelogs and think about what changed... The "the update caused the issue" statements are probably accurate less than 1% of the time someone posts it and never helpful as a mindset unless the changelog seems relevant to the issue. You almost insist that it is the cause (as per quoted sections from your post and replies) but the actual changes does not align with this perception. It almost creates an unwillingness to consider other reasons.

    We know what changed. You can too :) we show it inside CONTROL. From there its just a little logical thinking to convince yourself how likely it is to be a valid possibility versus observation/observer bias combined with an unrelated problem. That's the smarter way to begin Troubleshooting.

    Direct link to Changelog: https://raw.githubusercontent.com/OpenBuilds/OpenBuilds-CONTROL/master/CHANGELOG.txt

    Random resets / disconnects are listed on the EMI FAQ page: docs:blackbox-4x:faq-emi [OpenBuilds Documentation] along with lots of mitigation strategies.
     
    David the swarfer likes this.
  12. donald k wilson

    Builder

    Joined:
    Nov 10, 2016
    Messages:
    49
    Likes Received:
    11
    Thank you for the links, as I have said repeatedly. I now have some solid information to work with in a couple of areas I did not know were even related issues
     
  13. donald k wilson

    Builder

    Joined:
    Nov 10, 2016
    Messages:
    49
    Likes Received:
    11
    Some additional information/update...

    Yesterday afternoon I ran an air cut, no power to spindle, no vacuum, no issues. This morning I turned the computer on, loaded openbuilds control, nothing else and got a waiting for usb prompt in the connection window. Unplugging the cable from the back of the machine I got the com1 default and plugging back in, the FTDI usb to serial com3 dropdown. This is the normal (for me) connection dropdown, com3 is my connection. I connected, ran another small file in air cut, again no issues.

    While openbuilds could not connect to com3 my other usb devices were working however... This leads me to wonder if I have a failing usb port.

    Time to test with another computer...
     
  14. Peter Van Der Walt

    Peter Van Der Walt OpenBuilds Team
    Staff Member Moderator Builder Resident Builder

    Joined:
    Mar 1, 2017
    Messages:
    14,871
    Likes Received:
    4,283
    I would recommend that as well

    If the port is missing from the dropdown - the port has "disappeared" (You also wouldn't see it in Device Manager at that point)
    EMI can cause the OS to shut a port down - so it might still have some basis in that - but it's weird that it showed Waiting - if COM1 was an "option" (onboard serial port of the PC, not usable, but still "waiting" means 0 ports. COM1 should be 1 port, COM1 and COM3 would be 2 ports. Almost looks like the computer didn't even have COM1 active initially)
     
    donald k wilson likes this.
  15. donald k wilson

    Builder

    Joined:
    Nov 10, 2016
    Messages:
    49
    Likes Received:
    11
    I got the waiting for usb status again at startup. I have opened device manager to see what was going on, I've attached 3 screenshots of the startup process looking at device manager usb ports.
    1, computer just turned on.
    IMG_1087.JPG
    2, usb cable unplugged from the rear of the computer
    69291250251__9FBEBE3A-BFFA-4A89-8FD4-C8E79737B552.JPG
    3, usb cable plugged back in
    . 69291268534__1AE0A6E6-943B-4431-857E-C74D8C06B7C7.JPG
     
    #15 donald k wilson, Dec 16, 2022
    Last edited: Dec 16, 2022
  16. Peter Van Der Walt

    Peter Van Der Walt OpenBuilds Team
    Staff Member Moderator Builder Resident Builder

    Joined:
    Mar 1, 2017
    Messages:
    14,871
    Likes Received:
    4,283
    - Do you have any custom macros?
    - Did you change the Theme between screenshot 1 and 2? Or does that happen automatically
    - Is this the original PC or the other one you mentioned you would be testing with?
    - Press ctrl+shift+i and click the Console tab in Devtools. Screenshot any errors - open it up right after starting
     
  17. donald k wilson

    Builder

    Joined:
    Nov 10, 2016
    Messages:
    49
    Likes Received:
    11
    right after this I was able to run the complete cut file that stalled in the beginning of this post, no issues.
     
  18. donald k wilson

    Builder

    Joined:
    Nov 10, 2016
    Messages:
    49
    Likes Received:
    11
    This is not working for me, I start openbuilds control, push ctrl+shift+i and the program opens normally. No devtools or console anywhere to be seen.
    I have tried right after starting and after the program has opened, same result.
    I'm missing something... ;-)

    FYI, no com glitches opening and reopening OBC this time, still the same computer and usb cable.
     
  19. donald k wilson

    Builder

    Joined:
    Nov 10, 2016
    Messages:
    49
    Likes Received:
    11
    This happened just now after starting OBC successfully during a couple of jog commands to set up xyz zero for a new file.
    Nothing running but the stepper motors.
    same computer and usb cable.
    I notices a "bumping" sound coming from the motors just about the same time I lost the comm port, repeats 4 or 5 times, it does the same thing when I turn on the power supply to the stepper motors.
    I am now going to swap the usb cable for another with an rfi clip on it and try again... Screenshot (8).png
     
  20. donald k wilson

    Builder

    Joined:
    Nov 10, 2016
    Messages:
    49
    Likes Received:
    11
    New cable, smooth sailing on a long file, cutting with ungrounded dust collection, that might be next...
    screen shot shows port closed, I did that. Screenshot (9).png
     
  21. donald k wilson

    Builder

    Joined:
    Nov 10, 2016
    Messages:
    49
    Likes Received:
    11
    Here is the screenshot from devtools/console, I finally got it to open... Screenshot (10).png
     
  22. donald k wilson

    Builder

    Joined:
    Nov 10, 2016
    Messages:
    49
    Likes Received:
    11
    Another observation, I am only getting one "bump" from the steppers when I turn on the power supply with the new usb cable...?
     
  23. Peter Van Der Walt

    Peter Van Der Walt OpenBuilds Team
    Staff Member Moderator Builder Resident Builder

    Joined:
    Mar 1, 2017
    Messages:
    14,871
    Likes Received:
    4,283
    No errors in there, but keep an eye on it when you experience issues. In the same screenshot we see a list of ports so the "problem" was not preset... we want to see the console when its showing Waiting in case theres any errors there

    Pretty normal, as the computer runs port scans, it triggers the DTR pin of the serial port, causing controller to reset.

    Better USB cable still a good call for other EMI related potential issues. Filter (ferrite clip) alone won't make much difference to initial connection but will protect against high frequency noise

    That's a bad idea though, throw a Earth cable down the hose please - when the charges build up in the hose, order of several kilovolt, guess what happens when it discharges into nearby wiring, making its way down the line and possibly frying a computer USB port or USB chipset? If you see what I am thinking - unearthed Dust hose causes high voltage discharge spikes into your electronics. At best causing a disconnect, at worst already caused damage to your Computer causing intermittent symptoms (not unlikely)

    The "bump" is when the OS scan/opens the ports - triggering DTR (Data Terminal Ready) causing Grbl to reset. Pretty expected. Number of bumps is not really a clue to anything, but you'd have to watch the OSs USB stack closer to see if the bad cable had trouble enumerating causing multiple attempts.

    Some of last weeks questions have gone unanswered:
    And new question:
    So far we've by wayside discovered you aren't used the USB cable that came with BlackBox, but rather some other one that doesn't even have ferrites, we've discovered your hidden ungrounded dust hose. Lets save some time, tell us what else you have that is not up to the standard we expect from someone who'd have followed our docs :)
     
  24. donald k wilson

    Builder

    Joined:
    Nov 10, 2016
    Messages:
    49
    Likes Received:
    11
    No macros, I changed the theme for visibility, same pc, grounding cable for dust is next.
    I have not experienced the port closing since I switched the cable, I'll open the dev tool window if it happens again.
    I went with a longer usb cable to get my computer away from the cnc machine, the black box is mounted on the gantry and probably should be moved away? I was trying to minimize the bundle of com wires that had to travel to the computer.
    Spindle power and stepper motor cables are in the same bundle on the gantry, prob should be separated?
    Aside from the links you have already provided what else should I be looking at and I can tell you or correct whatever else needs attention.
    I set the machine up based on information available in the assembly videos, thought I was sticking pretty close to SOP, maybe not...

    Thank You!
     
  25. Peter Van Der Walt

    Peter Van Der Walt OpenBuilds Team
    Staff Member Moderator Builder Resident Builder

    Joined:
    Mar 1, 2017
    Messages:
    14,871
    Likes Received:
    4,283
    Yes, shorter USB cables are the way to go, as recommended

    Probably goes hand-in-hand with my earlier (unanswered) question then?
    So, you have a VFD Spindle, not our standard RoutER11? Any other kind of "not the usual" parts (ie from 3rd parties, or not included in the bundles) that I need to know about? The more you tell, the more we can help?
     
  26. donald k wilson

    Builder

    Joined:
    Nov 10, 2016
    Messages:
    49
    Likes Received:
    11
    Huanyang HY02D211B 2.2 vfd bundled with a chinese watercooled 1/2" er20 spindle.
    As I stated earlier, the vfd is about 3' away from the computer and cnc machine.
    Your black box and LRS-350-24 power supply.
    Black box is mounted on the gantry riser
    The cable bundle running right next to it and it sits about 6" above a stepper motor.
    Power cable is not shielded, that is an upgrade on the list.
    My machine is all OB components from scratch, not a kit. Very happy with the chassis after several modifications.
    two belt drive steppers running 2 1500 c beam y axis
    one threaded rod stepper running 1000 c beam x axis, gantry
    one threaded rod stepper running 500 c beam z axis, gantry
    All nema 23 motors, all assembled and wired according to instructions in your videos.
    This machine has been in R+D for some time (close to a year as a side project) working out rigidity issues.
    I haven't clocked the machine time but there are easily a couple hundred hours of "test" time, I have never had an electronic problem until this just now.
    Freecad v20 with their curves workbench addon for the complex curvature involved in an archtop guitar plate. Freecad's path mode has a GRBL translator I use to post process for OB.
    This has all worked pretty well cad to cam so I have continued using it.
     
  27. Peter Van Der Walt

    Peter Van Der Walt OpenBuilds Team
    Staff Member Moderator Builder Resident Builder

    Joined:
    Mar 1, 2017
    Messages:
    14,871
    Likes Received:
    4,283
    Thanks for the details!

    Is it a moving gantry machine, or fixed gantry? If its moving gantry, the USB cable may be tugging at the port causing disconnects. Moving the BlackBox may overall be a good idea - long term vibrations etc are also considerations
     
  28. donald k wilson

    Builder

    Joined:
    Nov 10, 2016
    Messages:
    49
    Likes Received:
    11
    Moving gantry, we considered the tug factor hence the longer usb.
    Vibration was an issue until I got the acceleration dialed in. It is much better now but...
    All things considered, moving the black box is low hanging fruit and I have some com cable slack to play with...
     
    Peter Van Der Walt likes this.
  29. Peter Van Der Walt

    Peter Van Der Walt OpenBuilds Team
    Staff Member Moderator Builder Resident Builder

    Joined:
    Mar 1, 2017
    Messages:
    14,871
    Likes Received:
    4,283
    I meant tool on material kind of more extreme vibration - when it start doing real work
     
  30. donald k wilson

    Builder

    Joined:
    Nov 10, 2016
    Messages:
    49
    Likes Received:
    11
    I meant tool on material kind of more extreme vibration - when it start doing real work

    AHA, Yes I had problems with that in the beginning especially with 1/2" bits and harder wood. Much R+D to stiffen things up and a feeds+speeds program to help calculate feed rates, cut depths and step overs.
    A sharp bit and much smoother sailing.
    I kept an eye on the connections during the trials because of those concerns but never seemed to have a problem...
     

Share This Page

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice