tail -f /var/log/DMIT.log
6.57K subscribers
191 photos
6 files
124 links
Download Telegram
DMIT has observed that some customers' applications running pose serious security risks and are being exploited on a large scale. This has already impacted DMIT's network security.
Investigation and mitigation in progress.
DMIT proactive security detected that 2% of tenants were deploying potentially risky applications in the DMIT LAX data center. These tenants' applications sent significant large traffic to China several hours ago.

- DMIT detected abnormal traffic within the network 12 hours ago but took no action due to complexity.
- However, this potential risk factor was officially exploited 8 hours ago.
- DMIT will officially notify affected customers within 12 hours and require them to resolve the issue within 24 hours to avoid service interruption for a long time and abuse penalties.
- The relevant users will receive an email soon, and the VM will be immediately shut down to avoid the transfer being used up. (DMIT will not recover this transfer quota since it was due to the client's unsafe application.)

This vulnerable application cannot directly generate such a large amount of traffic, even though it is open to the internet. Exploiting this type of vulnerability requires real data and requests; it does not possess amplification characteristics.

DMIT initially speculates that attackers in restricted areas are exploiting this TCP 443 application design flaw on a large scale to make super-large-scale real and accepted requests to OpenAI.

Further details will be officially provided after the issue is resolved.
LAX.EB:
AS9929 backbone issue.
High packet loss was observed

LAX.Pro:
Due to the AS9929 issue, the inbound was also impacted. The overall quality reduced.
tail -f /var/log/DMIT.log
LAX.EB: AS9929 backbone issue. High packet loss was observed LAX.Pro: Due to the AS9929 issue, the inbound was also impacted. The overall quality reduced.
LAX.Pro:

China Unicom's inbound routing has been temporarily rerouted via the GTT network.
Latency and packet loss issues have currently returned to normal levels.

We will switch the routing back to the original path once AS9929 returns to normal operation.

Although the AS3257-AS4837 route has a bandwidth of only 200 Gbps and cannot guarantee zero packet loss like AS9929 premium routes during normal time, it currently represents a "better" choice, given that AS9929 is causing packet loss rates exceeding 30% on the China Unicom (CU) network.
tail -f /var/log/DMIT.log
LAX.Pro: China Unicom's inbound routing has been temporarily rerouted via the GTT network. Latency and packet loss issues have currently returned to normal levels. We will switch the routing back to the original path once AS9929 returns to normal operation.…
Update from CUG NOC:

Territory backbone fiber cut leads to major capacity loss.

There are 144 of 288 fibers waiting to be repaired.

The repair is in progress.

ETR: unknowns.

*DMIT Notes: This fiber is between the cable landing station and L3 devices. It is highly possible that it is not owned by CUA. The repair process and updates should come from the local fiber vendor.
Los Angeles Major issue:

Widespread underlying fiber-optic cable failures affecting Cogent and Zayo have led to reduced network stability in the Los Angeles area.
Due to nested leasing arrangements within US Tier-1 backbones, the impact has extended to GTT and Arelion.
Packet loss has been observed on routes from Los Angeles toward the East (including Europe); the DMIT NOC has temporarily removed some of the most severely affected routes.

Some AS174 and AS3257 single home networks are not able to be migrated for network reachablity reason.

https://tranzact.zayo.com/#!/networkStatus

https://ecogent.cogentco.com/network-status
tail -f /var/log/DMIT.log
Los Angeles Major issue: Widespread underlying fiber-optic cable failures affecting Cogent and Zayo have led to reduced network stability in the Los Angeles area. Due to nested leasing arrangements within US Tier-1 backbones, the impact has extended to GTT…
DMIT has isolated the issue, except AS174 and AS3257 single home network.

Nevertheless, Cogent's Los Angeles metropolitan area network is experiencing a severe outage, with some parts of the network unable to access the Internet at all.

Screenshot shows the reachability of DMIT's Cogent interface and the location of failure.

Since this issue has been isolated, traffic was divert to NTT, GSL, Arelion and GTT. DMIT will stop update the network status of Cogent.
Both inbount and outbount routes from Cogent are removed 8hrs ago, since it's unusable.
DMIT has a new status page
For information · 2026-09-10 18:49 UTC

Our new status page is live at [does.dmit.fail](https://does.dmit.fail).

The full text is on the status page.
🟢 CN2 GIA access point degradation in CoreSite LA2
Minor · Resolved · 2026-09-18 16:36 UTC

Resolved 2026-09-18 16:36 UTC
China Telecom stated the issue at their CoreSite LA location has been resolved. DMIT has reloaded the routes received from China Telecom CoreSite LA.

Monitoring 2026-09-18 09:05 UTC
DMIT has detected packet loss on the CN2 GIA access point in CoreSite LA2. Traffic has been rerouted to the DRT CN2 GIA access point. Network performance for all products is currently unaffected.
tail -f /var/log/DMIT.log
🟢 CN2 GIA access point degradation in CoreSite LA2 Minor · Resolved · 2026-09-18 16:36 UTC Resolved 2026-09-18 16:36 UTC China Telecom stated the issue at their CoreSite LA location has been resolved. DMIT has reloaded the routes received from China Telecom…
🟢 CN2 GIA access point degradation in CoreSite LA2
Minor · Resolved · 2026-09-18 16:36 UTC

China Telecom stated the issue at their CoreSite LA location has been resolved. DMIT has reloaded the routes received from China Telecom CoreSite LA.
Channel name was changed to «tail -f /var/log/DMIT.log»
Starting from October 15, 2026, customers who register through social media accounts will need to set a login password.
Service impact · 2026-10-14 23:34 UTC

Starting October 15, 2026, customers who registered through social accounts and have not set a password will be required to do so. A prompt to set a password will appear upon login, with a grace period of 7 to 30 days depending on registration date. Failure to set a password after the grace period will suspend some client area functions.

The full text is on the status page.
🟢 Scheduled upgrade for Tokyo region software and kernel
Minor · Completed · 2026-10-21 03:26 UTC

Completed 2026-10-21 03:26 UTC
Kernel and software updates have been completed for all nodes in Tokyo.

In progress 2026-10-19 08:00 UTC
Maintenance has started.

Scheduled 2026-10-19 08:00 UTC
The software and kernel in the Tokyo region are one major version behind other regions.

A software and kernel upgrade is scheduled.

Downtime is up to 120 minutes per VM.
🟢 Scheduled maintenance: Online kernel patch for CVE-2026-99999
Minor · Completed · 2026-10-21 03:31 UTC

Completed 2026-10-21 03:31 UTC
All nodes have been hot-patched for CVE-2026-99999.

In progress 2026-10-19 08:00 UTC
Maintenance has started.

Scheduled 2026-10-19 08:00 UTC
DMIT will deploy a self-compiled online kernel patch to mitigate CVE-2026-99999 to all host nodes across all regions between October 19 and October 23, 2026.

Downtime is not anticipated, with the exception of the Tokyo region.

Should a reboot be necessary, virtual machines may experience up to 120 minutes of downtime.