|It has been a tough week for Internet connectivity in Lebanon. After two national Internet blackouts on the IMEWE cable, Lebanese traffic was moved onto the CADMOS submarine cable to reach international carriers via Cyprus. With this backup in place, and with substantial additional capacity brought online to reduce congestion, just over 70% of the country’s networks (prefixes) were brought back online.
In an example of engineering under pressure, Lebanese Telecoms Minister Nicholas Sehnaoui personally flew to Cyprus and met with the Cyprus Telecommunications Authority. The teams (pictured right) then collaborated to find a viable solution.
Crippling Internet Blackouts Reveal Dependence on IMEWE
Earlier this week, we reported on an outage due to cable maintenance on the IMEWE submarine cable. This incident took down nearly all international Internet access for Lebanon for 3 hours. Then on Wednesday, at approximately the same time of day (16:14 UTC), IMEWE was cut 50 kilometers from the coast of Alexandria in Egypt, once again interrupting connectivity country-wide. In each incident we observed minor network outages and routing instabilities in Egypt, Saudi Arabia, UAE and Pakistan (the other countries where IMEWE lands). None of those countries has the same reliance on a single cable, so they were all able to automatically and relatively seamlessly divert traffic to other routes.
Restoration Through Cyprus
At around 07:45 UTC today, we saw latencies into Lebanon improve as traffic started to flow via Cyprus. When viewing the outage and today’s restoration in traceroute data, we can see various Lebanese ISPs shifting through their Internet providers as they try to maintain service. Below we analyze these changes for a few of the main ISPs operating in country. It’s interesting to note that different providers seem to have moved their traffic onto the new alternative route at slightly different times, and not all at once.
Internet provider TerraNet shifts from Cogent and Level 3 over the IMEWE cable onto SatGate and followed by Tata. (Like SatGate, Tata’s service is also probably via satellite given the high median latencies (upper graph).) At 06:40 UTC, TerraNet was one of the first providers to move traffic to the Cyprus link.
|IDM is a bit of an enigma, going entirely offline following the cable break, losing Level 3 and Cogent. They then return with Telenor satellite service and some Level 3 service, presumably over the CADMOS link, and presumably heavily congested. (Note the higher median latencies.) Like the others, at about 07:40 UTC IDM gets its lower-latency service from Cyprus and begins to shed the higher-latency services.|
Lebanese ISP Sodetel begins with service primarily from Level 3 (AS3356) and Cogent (AS174) over IMEWE. However, once IMEWE becomes unavailable, traffic shifts to satellite providers Satgate (AS30721) and SIDUS (AS52036) and median latencies from around the world increase dramatically. At about 07:45 UTC, we see Cyprus Telecom Authority (AS6866) appear as a preferred upstream provider, and median latencies across the CADMOS cable drop back to values similar to those observed before the IMEWE outage.
Cyberia starts with transit primarily from Level 3, spends a few hours with no service, and then moves onto high-latency satellite service from VICUS (AS41589). Approximately ten minutes after Sodetel, at 07:55 UTC, Cyberia also begins to route traffic through Cyprus, resulting in significantly lower median latencies.
Having put together what appears to be a very workable interim solution with the help of the Cyprus Telecommunications Authority, the next step will be to see whether Sehnaoui and his team in Lebanon can keep this momentum going. Like the other countries in the region, Lebanon really needs multiple independent high-capacity paths (ideally through multiple competing Lebanese providers). Perhaps the Cyprus solution will be workable in the longer run; perhaps calmer days in Syria will someday enable a second, terrestrial path that accesses some of Turkey’s ample, high-bandwidth connectivity to Europe.
Lebanese bloggers have a wonderfully cynical, self-deprecating sense of humor about their outage-prone electrical grid and Internet service. Jokes aside, the continuing problems with maintaining stable, fast Internet are damaging to Lebanon’s reputation and make it harder to attract business investment. The CADMOS restoration success shows that Lebanon’s government and engineers deliver good results under fire. Will they now continue the regulatory and technical work to create a more resilient national Internet?
The evolution of Internet service in Bahrain could provide a useful role model.