Lessons Learned: Troubleshooting Azure Firewall, Application Gateway, and Route Tables

Azure networking can be a complex ecosystem, especially when dealing with Application Gateways (AppGW), Firewalls, and routing configurations. A recent troubleshooting journey within our community provided valuable insights into potential pitfalls and best practices when setting up Azure networking components. This blog captures the lessons learned from debugging an issue where traffic routing through a firewall resulted in HTTP 502 errors. 

The problem

The scenario involved deploying an Application Gateway (AppGW) and a backend server in the same Virtual Network (VNet) while routing traffic through an Azure Firewall using User Defined Routes (UDRs). According to Microsoft's documentation, this setup should work. However, after implementing the route tables, the system started returning HTTP 502 errors. 

Observations

• The AppGW could reach the backend server, but responses were not making their way back. The connection test from the Application Gateway showed successful connectivity, but real traffic failed. 
• Firewall logs did not show any blocked traffic, suggesting that responses never passed through it. 
•  A second backend pool in a separate VNet worked fine, indicating that the problem was specific to intra-VNet routing.

Community Troubleshooting and Investigation

The troubleshooting effort included multiple community members and involved analyzing several key areas: 

Checking Route Tables 

• Ensuring that UDRs were correctly configured and that more specific routes were overriding default ones.
•  Verifying that traffic was directed to the next hop (firewall's private IP) in both directions.
•  Confirming that routing intent in Virtual WAN (vWAN) was set to private traffic.

• Flow logs and effective routes on the VM NIC were reviewed to confirm how packets were moving. 
•  Checking whether traffic even reached the firewall, which in this case it seemed that it did not.
•  Asymmetric routing was considered as a potential issue.

DNS and Application Rules 

• Ensuring that custom DNS was correctly resolving backend IP addresses.  
•  Checking whether the firewall was interfering with name resolution.
•  Application Gateway's health probes and error logs were examined for anomalies.

Firewall Rule Order and Processing Logic 

• Microsoft's Firewall rule processing logic suggests that rules are applied in a specific order:  
•  NAT rules → Network rules → Application rules -  
•  A key observation was that no network rule existed for the required communication.
•  Instead, traffic was incorrectly matching an application rule, which allowed traffic to the front-end domain name of the AppGW
•  This resulted in a DNS lookup resolving to the AppGW itself, leading to a loop and the observed 502 errors.

The solution

After multiple rounds of debugging and an in-depth analysis of the routing configurations, we discovered that a crucial network rule was missing in the firewall. This rule should have explicitly allowed traffic from the AppGW subnet to the backend VM subnet. In its absence, the firewall defaulted to an application rule, resolving the frontend domain back to the AppGW and inadvertently causing a misroute. Adding the correct network rule to the firewall immediately resolved the issue.

Key Takeaways

It's important to ensure that network rules take precedence over application rules when managing inter-subnet traffic. A comprehensive review of firewall logs is crucial, as the presence or absence of logs can often point to a routing issue rather than a firewall block. In this case, the issue was identified when we observed that only the frontend domain was logged as an application hit, while none of the other Application Gateway domains were recorded in the application rule logs. Additionally, utilizing effective routes on NICs is essential to validate expected traffic flow. DNS resolution behavior should be carefully assessed, especially when firewalls are responsible for name lookups. Lastly, if Azure Virtual WAN (vWAN) is in use, it’s important to understand how it interacts with routing intent and firewall rules.

Final Thoughts 

This experience reinforced how interconnected Azure networking components are and how misconfigurations at one layer can lead to unexpected behavior at another. Troubleshooting involved systematically eliminating potential causes, leveraging flow logs, route tables, and rule processing logic to arrive at the solution. A special thanks to all contributors who engaged in debugging this issue, demonstrating the power of collaborative problem-solving in cloud networking! For more information on Azure Firewall rule processing, refer to Microsoft's official documentation. 

Happy Troubleshooting!