posted 06-17-2021 08:09 AM            

Operations Underway to Restore Payload Computer on NASA's Hubble Space Telescope

NASA is working to resolve an issue with the payload computer on the Hubble Space Telescope.

The computer halted on Sunday, June 13, shortly after 4 p.m. EDT. After analyzing the data, the Hubble operations team is investigating whether a degrading memory module led to the computer halt.

The team is preparing to switch to one of several backup modules on Wednesday, June 16. The computer will then be allowed to run for approximately one day to verify that the problem has been solved. The team would then restart all science instruments and return the telescope to normal science operations.

The purpose of the payload computer is to control and coordinate the science instruments onboard the spacecraft. After the halt occurred on Sunday, the main computer stopped receiving a "keep-alive" signal, which is a standard handshake between the payload and main spacecraft computers to indicate all is well. The main computer then automatically placed all science instruments in a safe mode configuration. Control center personnel at NASA's Goddard Space Flight Center in Greenbelt, Maryland restarted the computer on Monday, June 14, but it soon experienced the same problem.

The payload computer is a NASA Standard Spacecraft Computer-1 (NSSC-1) system built in the 1980s. It is part of the Science Instrument Command and Data Handling module, which was replaced during the last astronaut servicing mission in 2009. The module has various levels of redundancy which can be switched on to serve as the primary system when necessary.

posted 06-18-2021 12:57 PM               

Operations Continue to Restore Payload Computer on NASA's Hubble Space Telescope

NASA continues to work on resolving an issue with the payload computer on the Hubble Space Telescope. The operations team will be running tests and collecting more information on the system to further isolate the problem. The science instruments will remain in a safe mode state until the issue is resolved. The telescope itself and science instruments remain in good health.

The computer halted on Sunday, June 13. An attempt to restart the computer failed on Monday, June 14. Initial indications pointed to a degrading computer memory module as the source of the computer halt. When the operations team attempted to switch to a back-up memory module, however, the command to initiate the backup module failed to complete. Another attempt was conducted on both modules Thursday evening to obtain more diagnostic information while again trying to bring those memory modules online. However, those attempts were not successful.

The payload computer is a NASA Standard Spacecraft Computer-1 (NSSC-1) system built in the 1980s that is located on the Science Instrument Command and Data Handling unit. The computer's purpose is to control and coordinate the science instruments and monitor them for health and safety purposes. It is fully redundant in that a second computer, along with its associated hardware, exists on orbit that can be switched over to in the event of a problem. Both computers can access and use any of four independent memory modules, which each contain 64K of Complementary Metal-Oxide Semiconductor (CMOS) memory. The payload computer uses only one memory module operationally at a time, with the other three serving as backups.

posted 06-22-2021 11:49 PM               

Testing Underway to Identify Issue and Restore Payload Computer on NASA's Hubble Space Telescope

NASA continues to work to resolve a problem with the Hubble Space Telescope payload computer that halted on June 13. After performing tests on several of the computer's memory modules, the results indicate that a different piece of computer hardware may have caused the problem, with the memory errors being only a symptom.

The operations team is investigating whether the Standard Interface (STINT) hardware, which bridges communications between the computer's Central Processing Module (CPM) and other components, or the CPM itself is responsible for the issue. The team is currently designing tests that will be run in the next few days to attempt to further isolate the problem and identify a potential solution.

This step is important for determining what hardware is still working properly for future reference. If the problem with the payload computer can't be fixed, the operations team will be prepared to switch to the STINT and CPM hardware onboard the backup payload computer. The team has conducted ground tests and operations procedure reviews to verify all the commanding required to perform that switch on the spacecraft.

If the backup payload computer's CPM and STINT hardware is turned on, several days will be required to assess the computer performance and restore normal science operations. The backup computer has not been powered on since its installation in 2009; however, it was thoroughly tested on the ground prior to installation on the spacecraft.

The payload computer is a NASA Standard Spacecraft Computer-1 (NSSC-1) system built in the 1980s that is located on the Science Instrument Command and Data Handling (SI C&DH) unit. After 18 years on orbit, the original SI C&DH experienced a failure in 2008 that delayed the final servicing mission to Hubble while a replacement was prepared for flight. In May 2009, STS-125 was launched and the astronauts installed the existing unit.

The replacement contains original hardware from the 1980s with four independent 64K memory modules of Complementary Metal-Oxide Semiconductor (CMOS) memory. Only one memory module is used operationally, with the other three serving as backups. All four modules can be used and accessed from either of the redundant payload computers.

posted 06-25-2021 03:54 PM               

NASA Completes Additional Tests to Diagnose Computer Problem on Hubble Space Telescope

NASA is continuing to diagnose a problem with the payload computer on the Hubble Space Telescope after completing another set of tests on June 23 and 24. The payload computer halted on June 13 and the spacecraft stopped collecting science data. The telescope itself and its science instruments remain in good health and are currently in a safe configuration.

The spacecraft has two payload computers, one of which serves as a backup, that are located on the Science Instrument and Command and Data Handling (SI C&DH) unit. There are various pieces of hardware which make up both payload computers, including but not limited to:

• a Central Processing Module (CPM), which processes the commands that coordinate and control the science instruments

• a Standard Interface (STINT), which bridges communications between the computer's CPM and other components

• a communications bus, which contains lines that pass signals and data between hardware

• and one active memory module, which stores operational commands to the instruments. There are three additional modules which serve as backups.

Additional tests performed on June 23 and 24 included turning on the backup computer for the first time in space. The tests showed that numerous combinations of these hardware pieces from both the primary and backup payload computer all experienced the same error - commands to write into or read from memory were not successful.

Since it is highly unlikely that all individual hardware elements have a problem, the team is now looking at other hardware as the possible culprit, including the Command Unit/Science Data Formatter (CU/SDF), another module on the SI C&DH. The CU formats and sends commands and data to specific destinations, including the science instruments. The SDF formats the science data from the science instruments for transmission to the ground. The team is also looking at the power regulator to see if possibly the voltages being supplied to hardware are not what they should be. A power regulator ensures a steady constant voltage supply. If the voltage is out of limits, it could cause the problems observed.

Over the next week, the team will continue to assess hardware on the SI C&DH unit to identify if something else may be causing the problem. If the team determines the CU/SDF or the power regulator is the likely cause, they will recommend switching to the backup CU/SDF module and the backup power regulator.

posted 07-01-2021 10:30 PM               

NASA Preparing for Procedures to Turn On Backup Hardware on the Hubble Space Telescope

NASA is taking additional steps to investigate the Hubble Space Telescope's payload computer issue that began on June 13, suspending science observations. In parallel with the investigation, NASA is preparing and testing procedures to turn on backup hardware onboard the spacecraft. The telescope itself and science instruments remain healthy and in a safe configuration.

The source of the computer problem lies in the Science Instrument Command and Data Handling (SI C&DH) unit, where the payload computer resides. A few hardware pieces on the SI C&DH could be the culprit(s).

The team is currently scrutinizing the Command Unit/Science Data Formatter (CU/SDF), which sends and formats commands and data. They are also looking at a power regulator within the Power Control Unit, which is designed to ensure a steady voltage supply to the payload computer's hardware. If one of these systems is determined to be the likely cause, the team must complete a more complicated operations procedure to switch to the backup units. This procedure would be more complex and riskier than those the team executed last week, which involved switching to the backup payload computer hardware and memory modules. To switch to the backup CU/SDF or power regulator, several other hardware boxes on the spacecraft must also be switched due to the way they are connected to the SI C&DH unit.

Over the next week or so, the team will review and update all of the operations procedures, commands and other related items necessary to perform the switch to backup hardware. They will then test their execution against a high-fidelity simulator.

The team performed a similar switch in 2008, which allowed Hubble to continue normal science operations after a CU/SDF module failed. A servicing mission in 2009 then replaced the entire SI C&DH unit, including the faulty CU/SDF module, with the SI C&DH unit currently in use.

posted 07-09-2021 10:25 AM               

Successful Completion of Multi-Day Test

NASA successfully completed a test of procedures that would be used to switch to backup hardware on Hubble in response to the payload computer problem. This switch could occur next week after further preparations and reviews.

posted 07-15-2021 10:42 AM               

NASA Identifies Possible Cause of Hubble Computer Problem

NASA has identified the possible cause of the payload computer problem that suspended Hubble Space Telescope science operations on June 13. The telescope itself and science instruments remain healthy and in a safe configuration.

The payload computer resides in the Science Instrument Command and Data Handling (SI C&DH) unit. It controls, coordinates, and monitors Hubble's science instruments. When the payload computer halted, Hubble's science instruments were automatically placed into a safe configuration. A series of multi-day tests, which included attempts to restart and reconfigure the computer and the backup computer, were not successful, but the information gathered from those activities has led the Hubble team to determine that the possible cause of the problem is in the Power Control Unit (PCU).

The PCU also resides on the SI C&DH unit. It ensures a steady voltage supply to the payload computer's hardware. The PCU contains a power regulator that provides a constant five volts of electricity to the payload computer and its memory. A secondary protection circuit senses the voltage levels leaving the power regulator. If the voltage falls below or exceeds allowable levels, this secondary circuit tells the payload computer that it should cease operations. The team's analysis suggests that either the voltage level from the regulator is outside of acceptable levels (thereby tripping the secondary protection circuit), or the secondary protection circuit has degraded over time and is stuck in this inhibit state.

Because no ground commands were able to reset the PCU, the Hubble team will be switching over to the backup side of the SI C&DH unit that contains the backup PCU. All testing of procedures for the switch and associated reviews have been completed, and NASA management has given approval to proceed. The switch will begin Thursday, July 15, and, if successful, it will take several days to completely return the observatory to normal science operations.

The team performed a similar switch in 2008, which allowed Hubble to continue normal science operations after a Command Unit/Science Data Formatter (CU/SDF) module, another part of the SI C&DH, failed. A servicing mission in 2009 then replaced the entire SI C&DH unit, including the faulty CU/SDF module, with the SI C&DH unit currently in use.

posted 07-15-2021 02:29 PM               

NASA Begins Switch to Backup Spacecraft Hardware

Today, NASA began a switch to backup spacecraft hardware on Hubble in response to an ongoing problem with its payload computer. This will be a multi-day event. If successful, the next step will be for science instruments to be brought back into operation.

posted 07-16-2021 12:01 PM               

NASA Successfully Switches to Backup Hardware on Hubble Space Telescope

NASA has successfully switched to backup hardware on the Hubble Space Telescope, including powering on the backup payload computer, on July 15. The switch was performed to compensate for a problem with the original payload computer that occurred on June 13 when the computer halted, suspending science data collection.

The switch included bringing online the backup Power Control Unit (PCU) and the backup Command Unit/Science Data Formatter (CU/SDF) on the other side of the Science Instrument and Command & Data Handling (SI C&DH) unit. The PCU distributes power to the SI C&DH components, and the CU/SDF sends and formats commands and data. In addition, other pieces of hardware onboard Hubble were switched to their alternate interfaces to connect to this backup side of the SI C&DH. Once these steps were completed, the backup payload computer on this same unit was turned on and loaded with flight software and brought up to normal operations mode.

The Hubble team is now monitoring the hardware to ensure that everything is working properly. The team has also started the process for recovering the science instruments out of their safe mode configuration. This activity is expected to take more than a day as the team runs various procedures and ensures the instruments are at stable temperatures. The team will then conduct some initial calibration of the instruments before resuming normal science operations.

posted 07-17-2021 10:19 AM               

NASA Returns Hubble Space Telescope to Science Operations

NASA has returned the science instruments on the Hubble Space Telescope to operational status, and the collection of science data will now resume. This will be the first science data collected since the payload computer experienced a problem on June 13, which placed the instruments in a safe configuration and suspended science operations.

"Hubble is an icon, giving us incredible insight into the cosmos over the past three decades," said NASA Administrator Bill Nelson. "I'm proud of the Hubble team, from current members to Hubble alumni who stepped in to lend their support and expertise. Thanks to their dedication and thoughtful work, Hubble will continue to build on its 31-year legacy, broadening our horizons with its view of the universe."

The first observation is scheduled for Saturday afternoon after some instrument calibrations are completed. Most observations missed while science operations were suspended will be rescheduled for a later date.

The Hubble team has been investigating the cause of the payload computer problem since it first occurred. On July 15, the team switched the spacecraft to backup hardware.

NASA anticipates that Hubble will last for many more years and will continue making groundbreaking observations, working in tandem with other space observatories including the James Webb Space Telescope to further our knowledge of the cosmos.

posted 07-19-2021 05:13 PM               

Hubble Returns to Full Science Observations and Releases New Images

NASA's Hubble Space Telescope is back in business, exploring the universe near and far. The science instruments have returned to full operation, following recovery from a computer anomaly that suspended the telescope's observations for more than a month.

Above: These images, from a program led by Julianne Dalcanton of the University of Washington in Seattle, demonstrate Hubble's return to full science operations. [Left] ARP-MADORE2115-273 is a rarely observed example of a pair of interacting galaxies in the southern hemisphere. [Right] ARP-MADORE0002-503 is a large spiral galaxy with unusual, extended spiral arms. While most disk galaxies have an even number of spiral arms, this one has three. (Science: NASA, ESA, STScI, Julianne Dalcanton [UW] Image processing: Alyssa Pagan [STScI])

Science observations restarted the afternoon of Saturday, July 17. The telescope's targets this past weekend included the unusual galaxies shown in the images above.

"I'm thrilled to see that Hubble has its eye back on the universe, once again capturing the kind of images that have intrigued and inspired us for decades," said NASA Administrator Bill Nelson. "This is a moment to celebrate the success of a team truly dedicated to the mission. Through their efforts, Hubble will continue its 32nd year of discovery, and we will continue to learn from the observatory's transformational vision."

These snapshots, from a program led by Julianne Dalcanton of the University of Washington in Seattle, feature a galaxy with unusual extended spiral arms and the first high-resolution glimpse at an intriguing pair of colliding galaxies. Other initial targets for Hubble included globular star clusters and aurorae on the giant planet Jupiter.

Hubble's payload computer, which controls and coordinates the observatory's onboard science instruments, halted suddenly on June 13. When the main computer failed to receive a signal from the payload computer, it automatically placed Hubble's science instruments into safe mode. That meant the telescope would no longer be doing science while mission specialists analyzed the situation.

The Hubble team moved quickly to investigate what ailed the observatory, which orbits about 340 miles (547 kilometers) above Earth. Working from mission control at NASA's Goddard Space Flight Center in Greenbelt, Maryland, as well as remotely due to COVID-19 restrictions, engineers collaborated to figure out the cause of the problem.

Complicating matters, Hubble was launched in 1990 and has been observing the universe for over 31 years. To fix a telescope built in the 1980s, the team had to draw on the knowledge of staff from across its lengthy history.

Hubble alumni returned to support the current team in the recovery effort, lending decades of mission expertise. Retired staff who helped build the telescope, for example, knew the ins and outs of the Science Instrument and Command & Data Handling unit, where the payload computer resides – critical expertise for determining next steps for recovery. Other former team members lent a hand by scouring Hubble's original paperwork, surfacing 30- to 40-year-old documents that would help the team chart a path forward.

"That's one of the benefits of a program that's been running for over 30 years: the incredible amount of experience and expertise," said Nzinga Tull, Hubble systems anomaly response manager at Goddard. "It's been humbling and inspiring to engage with both the current team and those who have moved on to other projects. There's so much dedication to their fellow Hubble teammates, the observatory, and the science Hubble is famous for."

Together, team members new and old worked their way through the list of likely culprits, seeking to isolate the issue to ensure they have a full inventory for the future of which hardware is still working.

At first, the team thought the likeliest problem was a degrading memory module, but switching to backup modules failed to resolve the issue. The team then designed and ran tests, which involved turning on Hubble's backup payload computer for the first time in space, to determine whether two other components could be responsible: the Standard Interface hardware, which bridges communications between the computer's Central Processing Module and other components, or the Central Processing Module itself. Turning on the backup computer did not work, however, eliminating these possibilities as well.

The team then moved on to explore whether other hardware was at fault, including the Command Unit/Science Data Formatter and the Power Control Unit, which is designed to ensure a steady voltage supply to the payload computer's hardware. However, it would be more complicated to address either of these issues, and riskier for the telescope in general. Switching to these components' backup units would require switching several other hardware boxes as well.

"The switch required 15 hours of spacecraft commanding from the ground. The main computer had to be turned off, and a backup safe mode computer temporarily took over the spacecraft. Several boxes also had to be powered on that were never turned on before in space, and other hardware needed their interfaces switched," said Jim Jeletic, Hubble deputy project manager at Goddard. "There was no reason to believe that all of this wouldn't work, but it's the team's job to be nervous and think of everything that could go wrong and how we might compensate for it. The team meticulously planned and tested every small step on the ground to make sure they got it right."

The team proceeded carefully and systematically from there. Over the following two weeks, more than 50 people worked to review, update, and vet the procedures to switch to backup hardware, testing them on a high-fidelity simulator and holding a formal review of the proposed plan.

Simultaneously, the team analyzed the data from their earlier tests, and their findings pointed to the Power Control Unit as the possible cause of the issue. On July 15, they made the planned switch to the backup side of the Science Instrument and Command & Data Handling unit, which contains the backup Power Control Unit.

Victory came around 11:30 p.m. EDT July 15, when the team determined the switch was successful. The science instruments were then brought to operational status, and Hubble began taking scientific data once again on July 17. Most observations missed while science operations were suspended will be rescheduled.

This is not the first time Hubble has had to rely on backup hardware. The team performed a similar switch in 2008, returning Hubble to normal operations after another part of the Science Instrument and Command & Data Handling (SI C&DH) unit failed. Hubble's final servicing mission in 2009 – a much-needed tune-up championed by former U.S. Senator Barbara Mikulski – then replaced the entire SI C&DH unit, greatly extending Hubble's operational lifetime.

Since that servicing mission, Hubble has taken more than 600,000 observations, bringing its lifetime total to more than 1.5 million. Those observations continue to change our understanding of the universe.

"Hubble is in good hands. The Hubble team has once again shown its resiliency and prowess in addressing the inevitable anomalies that arise from operating the world's most famous telescope in the harshness of space," said Kenneth Sembach, director of the Space Telescope Science Institute (STScI) in Baltimore, Maryland, which conducts Hubble science operations. "I am impressed by the team's dedication and common purpose over the past month to return Hubble to service. Now that Hubble is once again providing unprecedented views of the universe, I fully expect it will continue to astound us with many more scientific discoveries ahead."

Hubble has contributed to some of the most significant discoveries of our cosmos, including the accelerating expansion of the universe, the evolution of galaxies over time, and the first atmospheric studies of planets beyond our solar system. Its mission was to spend at least 15 years probing the farthest and faintest reaches of the cosmos, and it continues to far exceed this goal.

"The sheer volume of record-breaking science Hubble has delivered is staggering," said Thomas Zurbuchen, associate administrator of NASA's Science Mission Directorate. "We have so much to learn from this next chapter of Hubble's life – on its own, and together with the capabilities of other NASA observatories. I couldn't be more excited about what the Hubble team has achieved over the past few weeks. They've met the challenges of this process head on, ensuring that Hubble's days of exploration are far from over."