Database Maintenance (Resolved)
  • Priority - High
  • Affecting System - Centova Cast v3
  • Update: The maintenance window has now closed. The Centova Cast v3 control panel is online.

    SERVICE AFFECTING DATABASE MAINTENANCE!

    ================================

    EVENT IMPACT: Centova Cast v3 control panel

    EVENT WINDOW: Feb 18th, 2019 9:00am CST

    ================================

    In order to maximize the speed of Centova Cast v3 scheduled database maintenance is required. Please be advised that next Monday morning, February 18th, the control panel may be unavailable at times during business hours. Best case scenario is sixty minutes offline. However, all streamers and stations need to be prepared in advance.

    Shoutcast and Icecast servers will remain online and active during this interim as an upgrade from MySQL 5.73 to MariaDB 10.3 takes place. Migration to the InnoDB storage engine should boost performance and no downtime is expected. Adjust your broadcasts accordingly.

    Thank you for your patronage!

  • Date - 02/18/2019 09:00 - 02/18/2019 17:00
  • Last Updated - 02/18/2019 20:10
Emergency Maintenance (Resolved)
  • Priority - Critical
  • Affecting Server - Tank (Centova Cast)
  • PLEASE READ! Emergency maintenance RESOLVED 09/07/10!

    You are receiving this email notification because your hosting service WAS being directly impacted.

    Yesterday morning at approximately 9:36 a.m. service was restored to the Tank server. This issue was specifically hard to troubleshoot due to the fact that there were really no symptoms or tell tale signs. All in all the RAM was swapped, kernel version updated, RAID array rebuilt, and motherboard replaced.

    Since the fall out, a day later, I have once again become confident in the machine and want to thank all of you for your patience. I understand how important uptime is to our clients. Hence, Monday morning when I was alerted of an outage event at 1:41 a.m., as well as Tuesday morning at 3:55 a.m., I stayed awake working diligently for the rest of day in order to prevent as little downtime as possible.

    Thanks again, truly,

    -Seth Cole

    RockandRollHosting.com - "Be Heard" [Established 2001]
    info@rockandrollhosting.com | www.rockandrollhosting.com

    -------------------------------------------------

    PLEASE READ! Emergency maintenance scheduled 09/07/10!

    You are receiving this email notification because your hosting service is being directly impacted.

    The past twenty four hours have been a technical whirlwind as I, personally, along with some of the datacenter level three admins have been giving forth our best efforts towards a permanent solution to the Tank server outages. The RAID array has been rebuilt, kernel version updated, and RAM replaced with upgraded firmware. We have done as much of this work as possible through the early a.m. hours (off peak) to prevent as little distraction as possible. However this morning that work has spilled over into daylight as we continue forward.

    The datacenter has a spare motherboard on location in the event that it is deemed necessary. Unfortunately this problem has been hard to diagnose leaving very few clues or error messages to aid us in the repair.

    Thanks for your patience,

    -Seth Cole

    RockandRollHosting.com - "Be Heard" [Established 2001]
    info@rockandrollhosting.com | www.rockandrollhosting.com

    -------------------------------------------------

    PLEASE READ! Emergency maintenance scheduled 09/06/10!

    You are receiving this message because your hosting service is being directly impacted. Over the past ten days our "Tank" server has crashed three times, causing nearly two hours of downtime, including once late last night. I have been burning the midnight oil since 1:41 a.m. CST. After further investigation it seems that this could be only one of two potential issues.

    1. Hardware Failure - Although the server is not reporting any disk errors, we are suspicious about the RAID array. Fortunately our data is redundant and not at risk but a replacement could cause further downtime.

    2. Kernel Panic - In mid August we upgraded to the newest released Linux kernel version (2.6.9-89.0.28.ELsmp) which ran seemingly fine for a few weeks, however well worth due diligence. To learn more: http://en.wikipedia.org/wiki/Kernel_panic

    A few moments ago I rebooted the server onto the previous kernel (kernel-2.6.9-89.0.25.EL). This was the most sound fix to attempt first because it only requires a reboot. We will continue to monitor the situation and update you briefly.

    Thanks for your patience,

    -Seth Cole

    RockandRollHosting.com - "Be Heard" [Established 2001]
    info@rockandrollhosting.com | www.rockandrollhosting.com

  • Date - 09/06/2010 08:13 - 09/08/2010 11:39
  • Last Updated - 09/08/2010 11:39
Critical Hardware Failure / File Corruption (Resolved)
  • Priority - Critical
  • Affecting Server - Army (Chicago-1)
  • At approximitaly 4 p.m. Friday afternoon I/O swapping on the "Army" server was causing significant wait times for several services; MySQL, Apache, Exim, etc. The server load continued to grow so a reboot was scheduled near 6 p.m., with only a brief downtime the problem appeared to be solved.

    Near 3 a.m. Saturday morning our monitors alerted us to a hard drive failure in the RAID 10 disk array on the "Army" server. RockandRollHosting.com takes every precaution to prevent downtime and all services continued to run albeit without the high performance we expect.

    9:00 a.m. Saturday morning a serious investigation to permanently resolve the "Army" issues went under way. After an upgrade to the latest kernel, and a firmware update for the RAID array, a file system corruption was discovered on the /home partition. The server was then taken offline for approximetely thirty minutes to an hour for a live CD file system check and repair. The issue subsided and the server was put back online. However once again the server load slowly began to rise. After a second file system check and repair Saturday afternoon the problem once again subsided.

    12:30 p.m. Sunday afternoon the problem began to reoccur. At that point it was decided that the only resolution was to unmount and mount the /home partition after reformatting the drive. The server was taken offline and quickly reformatted. Our daily local backups were on site and available. However 100 Gigabytes of content doesn't quickly restore itself. Finally @ 10:49 p.m. the server came back online. There was a MySQL permission error which caused a small amount of sites to remain down until only a few hours ago.

    100% of the back ups were restored with no data loss. This includes emails, files, databases, etc.

    If you have any questions or concerns please don't hesitate to contact us.

  • Date - 04/04/2010 09:03 - 04/07/2010 16:37
  • Last Updated - 04/05/2010 09:19
Emergency Maintenance: 02-21-2010 (Resolved)
  • Priority - High
  • Affecting Server - Army (Chicago-1)
  • EVENT IMPACT: Core network upgrades, expected periods of increased latency during route propagation

    EVENT DETAILS: On Sunday, February 21st, 2010 between 00:00 (Midnight) and 06:00 (6am) CST, we will be performing emergency maintenance to address stability and redundancy issues for a core network link which services the ALO01 datacenter.

    This maintenance is only significant to customers with services hosted on the army.rockandrollhosting.com server and no service disruptions are expected. However, because the maintenance involves core network components this maintenance window is being announced. Customers may notice periods of increased latency during this maintenance window as route updates are applied.

  • Date - 02/21/2010 00:00 - 09/07/2010 08:13
  • Last Updated - 02/19/2010 09:23
Annual UPS Maintenance (Resolved)
  • Priority - Medium
  • Affecting Server - Army (Chicago-1)
  • CoreSite personnel will be performing UPS 1A, 3A, 1B, 3B annual maintenance checks.  This will include the following: Transferring the site to generator power, placing UPS in bypass, placing UPS in maintenance bypass, shutting down the UPS, performing maintenance, and restoring the site. Alarms expected are as follows: Gen supplying load, Gen indicates an alarm condition, UPS on battery, UPS in bypass indicates an alarm condition, UPS indicates an alarm condition, and ATS in emergency.  There will be no affect on the end customer.  Tech personnel will be onsite and working with CoreSite to ensure a successful maintenance.  If you have any questions regarding this maintenance, please open a support ticket via the Client Portal.

  • Date - 02/03/2010 00:00 - 02/03/2010 17:00
  • Last Updated - 02/19/2010 09:23
67.212.189.10 Scheduled Downtime - 3:30 a.m. (Resolved)
  • Priority - Critical
  • Affecting Server - Nuke (Chicago-3)
  • PLEASE READ THIS NOTIFICATION AS IT MAY EFFECT YOUR STREAMING SERVICE!

    67.212.189.10 Scheduled Downtime - 3:30 a.m. CST Feb. 8th

    _________________________________________________

    The time has come for us to do some necessary upgrades on the "Nuke" server (IP - 67.212.189.10). These updates require between 30 - 60 minutes of downtime as well as a reboot. We have scheduled this during non-peak time hours on a Monday morning after the Superbowl to prevent as few people from noticing as possible. However we believe our clients should be aware of this downtime in advance.

    As always, thanks for your loyalty, patience, and understanding.

  • Date - 02/08/2010 00:00 - 02/23/2010 08:04
  • Last Updated - 02/07/2010 10:02
Kernel & OS Upgrade (Resolved)
  • Priority - High
  • Affecting Server - Nuke (Chicago-3)
  • As experienced web hosting professionals, we know how important keeping an up-to-date kernel is to the security and stability of our Linux web servers. It is because of this that we have always had a very stringent and ongoing focus on security.

    Our quality assurance system test has indicated that one of our servers (67.212.xxx.xx) is NOT currently running a supported kernel due to the amount of time between the last system update and the last full reboot.  As a result, in order to successfully apply this service upgrade, we will need to reboot the server into the latest supported kernel.  The update is scheduled to take place on Nov 7th starting at 12AM CST and no interruption of the server's existing functionality or stability is expected.  Furthermore, no intervention from yourself is required.  A support technician will be on site and supervising the reboot to ensure that the system comes back online as promptly as possible and with no issues.

  • Date - 11/07/2009 15:42 - 11/08/2009 13:45
  • Last Updated - 11/06/2009 15:47
Scheduled Maintenance (Resolved)
  • Priority - Medium
  • Affecting Server - Army (Chicago-1)
  • START DATE: 2 NOV 2009
    START TIME: 06:00 (6:00am) CENTRAL
     
    END DATE: 6 NOV 2009
    END TIME: 22:00 (10:00pm) CENTRAL
     
    WORK TYPE: Power
    WORK PURPOSE: Upgrade/Expansion
    IMPACT: 427 La Salle (CoreSite)

     
    EVENT DETAILS:
    CoreSite will be performing a full commissioning of all systems related to the second phase IDC installation located on the third floor of the building. While no impact is expected, CoreSite will have building, mechanical, and electrical engineers on site in addition to CoreSite Operations staff throughout the entire maintenance window. RockandRollHosting.com is coordinating closely with CoreSite to ensure all maintenance steps are performed to mitigate any risk to power delivery.
     
    If you have any questions or concerns about this maintenance notification, please open a support ticket via the Client Portal.

  • Date - 11/02/2009 00:00 - 11/04/2009 09:07
  • Last Updated - 10/28/2009 09:54
Power Failure (Resolved)
  • Priority - Critical
  • Affecting Server - Nest (Chicago-2)
  • On August 28th, at approximately 17:39 PDT there was a power failure in our Virginia data center. Local staff confirmed with the building engineers that all of the Liebert UPS modules were offline.

    The buildings electrical service company MC Dean was on site doing a non-intrusive infrared scan of all of the critical electrical equipment during a scheduled maintenance tonight. They reached a point in the maintenance that required the system to be put into maintenance bypass. When they transferred to bypass one of the feeders behind a static transfer switch shorted to ground causing the UPS system to go offline. After identifying the problem the cable was repaired and the system was re-energized at 18:28PM PST.

  • Date - 08/28/2009 00:00 - 08/28/2009 00:00
  • Last Updated - 08/29/2009 08:34
Perl Upgrade - v5.8.5 to v5.8.8 (Resolved)
  • Priority - Medium
  • Affecting Server - Army (Chicago-1)
  • Early this morning a module required by cPanel (Detector) was unable to complete an auto-install due to the fact that the Perl version on army.rockandrollhosting.com needs to be updated to v5.8.8. The system may not function fully properly until this update occurs. We are currently processing this upgrade and no downtime is expected. However cPanel maybe temporarily unavailable.

  • Date - 08/26/2009 14:39 - 08/26/2009 00:00
  • Last Updated - 08/26/2009 14:47

Server Summary

Below is a live snapshot of the ROCKHOST.COM Incorporated network. For an in depth third party report, including transparent operations history, please see our public Panopta page.

Server Name Online Server Load Uptime
Army (Chicago-1)
Buzz (Chicago-4)
Hive (Atlanta-D)
Moth (Salt Lake City-F)
Nest (Chicago-2)
Nuke (Chicago-3)