url-domain-reputation - Security Engineer building AI-driven defensive too

项目说明

Malicious URL & Domain Reputation Scoring Model

A URL and domain reputation system that scores links for malicious intent before a user clicks. This project mirrors what security gateways, email filters, and SOC enrichment systems do by analyzing URLs using lexical, DNS-like, and reputation features to classify links as benign, suspicious, or malicious.

How URL Reputation Works in Security Gateways

URL reputation is a critical component of modern security infrastructure. Security gateways, email filters, and web proxies use reputation systems to:

Pre-Click Protection

Email Security: Scan links in incoming emails before delivery, flagging or quarantining malicious URLs

Web Gateways: Block access to known malicious domains at the network edge

Browser Extensions: Warn users about suspicious links before they click

SOC Enrichment: Provide context for security analysts in incidents

The Reputation Pipeline

Integration Points

Secure Email Gateways (SEG): Microsoft Defender for Office 365, Mimecast, Proofpoint

Secure Web Gateways (SWG): Zscaler, Cisco Umbrella, Palo Alto

SIEM/SOAR: Splunk, Sentinel, Cortex XSOAR for incident enrichment

API Gateways: Custom applications checking URLs before processing

Why It Matters

Speed: Prevents users from reaching malicious sites

Scale: Automated protection for thousands of URLs per second

Accuracy: ML models detect patterns humans might miss

Consistency: Uniform protection across the organization

Feature Signals and Why They Matter

The system extracts 11 features from each URL to assess malicious intent:

1. URL Length (url_length)

What it measures: Total character count of the URL

Why it matters: Malicious URLs are often unusually long to:

Obfuscate the actual domain

Include tracking parameters

Bypass length-based filters

Hide malicious keywords in long strings

Typical ranges:

Benign: 20-60 characters

Suspicious: 60-100 characters

Malicious: 100+ characters

2. Path Length (path_length)

What it measures: Character count of the URL path (after domain)

Why it matters: Long paths often indicate:

Phishing pages mimicking legitimate structures

Directory tra attempts

Obfuscated landing pages

Redirection chains

3. Number of Dots (num_dots)

What it measures: Count of dot characters in the URL

Why it matters: Excessive dots suggest:

Subdomain abuse (e.g., secure.login.fake.com)

Domain hiding techniques

Typosquatting (e.g., gooogle.com)

Internationalized domain name (IDN) attacks

4. Number of Hyphens (num_hyphens)

What it measures: Count of hyphen characters in the URL

Why it matters: Hyphens are commonly used in:

Lookalike domains (e.g., secure-paypal-login.com)

Keyword stuffing for SEO manipulation

Separating words in suspicious domains

Bypassing keyword filters

5. Number of Digits (num_digits)

What it measures: Count of numeric characters in the URL

Why it matters: High digit counts indicate:

Randomized domain names (DGAs - Domain Generation Algorithms)

Tracking IDs and session tokens

IP address usage instead of domain names

Obfuscation techniques

6. HTTPS Usage (uses_https)

What it measures: Binary flag (1 if HTTPS, 0 if HTTP)

Why it matters: While HTTPS doesn't guarantee safety:

Legitimate services increasingly use HTTPS

Malicious sites also use HTTPS (free certificates)

HTTP-only sites are increasingly suspicious

Context matters when combined with other signals

7. IP Address in URL (has_ip)

What it measures: Binary flag (1 if URL contains IP address)

Why it matters: Direct IP addresses are highly suspicious because:

Legitimate services use domain names

IP addresses bypass domain reputation systems

Common in phishing and malware distribution

Indicates potential infrastructure compromise

8. Subdomain Count (subdomain_count)

What it measures: Number of subdomain levels

Why it matters: Excessi suggest:

Subdomain abuse for hosting malicious content

Free subdomain services (e.g., github.io, wordpress.com)

Attempting to appear legitimate with corporate-like structures

Bypassing domain blacklists

9. Shannon Entropy (entropy)

What it measures: Randomness of character distribution (0-8 bits)

Why it matters: High entropy indicates:

Randomized strings (DGAs, tracking IDs)

Obfuscation techniques

Machine-generated URLs

Low human readability

Typical ranges:

Benign: 2.5-3.5 bits

Suspicious: 3.5-4.5 bits

Malicious: 4.5+ bits

10. Suspicious Word Count (suspicious_word_count)

What it measures: Count of words from a suspicious keyword list

Keywords: login, , update, secure, account, free, bonus, gift

Why it matters: These words are commonly used in:

Phishing attacks creating urgency or trust

Social engineering manipulation

Fake authentication pages

Too-good-to-be-true offers

11. High-Risk TLD (high_risk_tld)

What it measures: Binary flag (1 if TLD is in high-risk list)

High-risk TLDs: ru, cn, tk, ml, ga, cf

Why it matters: These TLDs are associated with:

Lower registration costs (attracting abuse)

Less stringent requirements

Higher historical rates of malicious hosting

Geographic regions with different threat profiles

Risk Score Interpretation

The system produces a risk score from 0-100 based on the model's confidence and classification.

Score Calculation

Risk Thresholds

| Score Range | Classification | Interpretation | Recommended Action |
|-------------|----------------|----------------|-------------------|
| 0-30 | Low Risk | Likely benign | Allow with standard logging |
| 31-50 | Medium Risk | Suspicious characteristics | Allow with warning or additional |
| 51-70 | High Risk | Strong malicious indicators | Block or require user confirmation |
| 71-100 | Critical Risk | Very likely malicious | Block immediately and alert security team |

Example Scores

| URL | Label | Confidence | Risk Score | Interpretation |
|-----|-------|------------|------------|----------------|
| | benign | 0.95 | 1.5 | Very low risk - legitimate Google domain |
| | malicious | 0.92 | 92 | Critical risk - obvious phishing attempt |
| | suspicious | 0.75 | 75 | High risk - suspicious offer domain |
| | benign | 0.98 | 0.6 | Very low risk - legitimate Microsoft domain |
| | malicious | 0.88 | 88 | Critical risk - high-risk TLD with suspicious keywords |

Using Risk Scores in Practice

Email Security:

Score 0-30: Deli

Score 31-50: Deliver with warning banner

Score 51-70: Quarantine for review

Score 71-100: Block and notify security team

Web Gateway:

Score 0-30: Allow access

Score 31-50: Allow with user warning

Score 51-70: Block with o option

Score 71-100: Block completely

SOC Enrichment:

All scores: Log for analysis

Score 50+: Flag for analyst review

Score 70+: Create security incident

Explainability Approach

The system provides explainable decisions using feature importance from the RandomForest model.

Feature Importance

RandomForest models naturally provide feature importance scores indicating which features contributed most to the classification decision.

Top Features Output

Each scoring result includes the top 5 contributing features:

Why Explainability Matters

For Security Analysts:

Understand why a URL was flagged

model decisions

Identify false positi

Impro rules

For Users:

Trust the system's decisions

Learn to recognize malicious patterns

Make informed o decisions

For Compliance:

Audit trail of security decisions

Regulatory requirements for explainable AI

Incident in support

Interpreting Top Features

Example:

1. suspicious_word_count: Contains "secure", "login", "update" - classic phishing keywords
2. high_risk_tld: Uses .com (not high-risk in this case, but model learned pattern)
3. num_hyphens: Multiple hyphens separating suspicious words
4. entropy: High randomness indicating potential obfuscation
5. url_length: Longer than typical legitimate PayPal URLs

Limitations of Current Explainability

Global : Feature importance is global (model-wide), not local (per-URL)

Correlation: Doesn't show feature interactions or correlations

Direction: Doesn't indicate whether high or low contributed

Future Impro: SHAP , LIME, or permutation importance for local explanations

Batch Usage

The system supports both batch processing for analysis and real-time scoring for integration.

Batch Scoring

Use Cases:

Analyzing historical URL datasets

Periodic reputation audits

Training data preparation

Security research and threat hunting

How to Use:

Input Format (CSV):

Output Format (CSV):

Ad:

Process thousands of URLs efficiently

Generate reports for stakeholders

Build training datasets

Offline analysis without network dependencies

Real-Time Scoring

Use Cases:

Email gateway integration

Web proxy filtering

API endpoint for applications

Browser extension backends

How to Use (API):

Response:

Ad:

Instant protection for users

Integration with existing security infrastructure

Low latency (<100ms per URL)

Scalable with horizontal deployment

Performance Considerations

Batch:

Throughput: ~10,000 URLs/second (depends on hardware)

Latency: Not applicable (bulk processing)

Resource usage: CPU-intensi processing

Real-Time:

Throughput: ~100 URLs/second per instance

Latency: 50-100ms per URL

Resource usage: Constant baseline, spikes with traffic

Scaling:

Batch: Parallel processing across multiple cores/machines

Real-Time: Load balancing across multiple API instances

Limitations

No Live DNS Resolution

What it means: The system does not perform actual DNS lookups or network queries.

Implications:

Cannot detect IP-based reputation

Cannot check domain age or registration history

Cannot SSL certificate

Cannot detect fast-flux networks

Workaround: Integrate with external DNS reputation services (, Cisco Umbrella, etc.)

No Hosting Intelligence

What it means: The system does not analyze the actual hosting infrastructure.

Implications:

Cannot detect compromised legitimate domains

Cannot check hosting provider reputation

Cannot identify shared hosting abuse

Cannot detect CDN-based attacks

Workaround: Integrate with threat intelligence feeds and hosting reputation APIs

No Content Analysis

What it means: The system does not fetch or analyze the actual web page content.

Implications:

Cannot detect page content (phishing forms, malware downloads)

Cannot analyze Ja behavior

Cannot detect drive-by downloads

Cannot check for page obfuscation

Workaround: Integrate with web crawling and content analysis tools

Static Feature Set

What it means: The system uses a fixed set of 11 lexical features.

Implications:

May miss emerging attack patterns

Limited to URL structure analysis

Cannot adapt to new e techniques

Requires retraining to add new features

Workaround: Regular model retraining with new data, feature engineering iterations

Training Data Dependency

What it means: Model performance depends on the quality and representati of training data.

Implications:

May ha positives on unusual but legitimate URLs

May miss no patterns not in training data

Requires labeled datasets (benign/malicious/suspicious)

Performance degrades over time as attack techniques evolve

Workaround: Continuous monitoring, feedback loops, periodic retraining

No Geographic Context

What it means: The system does not consider the geographic location of the requester.

Implications:

Cannot detect anomalous access patterns

Cannot apply geo-based policies

Cannot detect foreign IP accessing domestic resources

Workaround: Integrate with IP geolocation and access control systems

Extensions

WHOIS Age Analysis

Concept: Check domain registration age and expiration.

Implementation:

Why it matters:

Newly registered domains are more likely malicious

Legitimate domains typically ha histories

Fast-flux attacks use short-li

Age is a strong reputation signal

Passive DNS Integration

Concept: Analyze historical DNS records and IP associations.

Implementation:

Why it matters:

Detect domain hijacking

Identify IP reputation through historical associations

Detect fast-flux networks

Track domain e over time

Threat Feed Integration

Concept: Incorporate external threat intelligence feeds.

Implementation:

Why it matters:

Le community intelligence

Detect known malicious URLs

Reduce false negatives

Provide context from multiple sources

SSL/TLS Certificate Analysis

Concept: Analyze SSL certificate metadata.

Implementation:

Why it matters:

Detect self-signed certificates

Identify certificate issuer reputation

Check certificate period

Detect certificate spoofing

Beha Analysis

Concept: Track URL behavior over time (clicks, redirects, etc.).

Implementation:

Why it matters:

Detect compromise of pre benign domains

Identify redirection abuse

Track campaign patterns

Enable proacti

Deep Learning Models

Concept: Replace or augment RandomForest with neural networks.

Implementation:

Why it matters:

Capture complex non-linear patterns

Learn from sequential data (character-le)

Better generalization to no

Automatic feature learning

Ensemble Methods

Concept: Combine multiple models for impro.

Implementation:

Why it matters:

Reduce model bias

Impro

Better handling of edge cases

Confidence through consensus

Installation

Prerequisites

Python 3.8 or higher

pip package manager

Setup

1. Clone the repository

2. Create en

3. Install dependencies

Usage

Training the Model

Train the RandomForest classifier on the provided dataset:

This will:

Load URLs from data/raw/urls.csv

Extract features for each URL

Train a RandomForest classifier

Save the model to src/models/url_rf.joblib

Save the label encoder to src/models/label_encoder.joblib

Print classification report with metrics

Batch Scoring

Score a CSV file of URLs:

This will:

Score all URLs in data/raw/urls.csv

Sa to data/processed/scored_urls.csv

Generate a markdown report at docs/examples/url_report.md

Real-Time API

Start the FastAPI server:

The API will be a at

#### API Endpoint

POST /score

- Request body: {"url": "
- Response: JSON with label, confidence, risk_score, and top_features

#### Example Request

#### Example Response

Project Structure

Model Performance

The RandomForest classifier is trained with the following hyperparameters:

n_estimators: 400 trees

random_state: 42 for reproducibility

n_jobs: -1 (use all a cores)

Expected Metrics

With a properly labeled and balanced dataset:

Accuracy: 85-95%

Precision: 80-90% (minimizing false positives)

Recall: 85-95% (minimizing false negatives)

F1-Score: 82-92%

Improving Performance

1. Increase training data: More labeled URLs impro generalization
2. Feature engineering: Add new features based on attack patterns
3. Hyperparameter tuning: Use grid search or random search
4. Ensemble methods: Combine multiple models
5. Regular retraining: Update model with new threat data

Contributing

Contributions are welcome. Please ensure:

New features are well-documented

Code follows existing style

Changes are tested with URL patterns

Performance impact is considered

License

This project is provided as-is for educational and organizational use.

Support

For issues, questions, or contributions, please open an issue on the GitHub repository.

Acknowledgments

This project demonstrates how machine learning can be applied to URL reputation analysis, mirroring techniques used in enterprise security gateways and email filters.

项目信息

开发语言：Python

Star 数：1

技术要点

该项目属于防护思路方向的开源资源，代码结构清晰，适合安全研究人员学习参考。项目围绕Security Engineer building AI-driven defensive tools: Zero-Trust access engines, URL reputation systems, and human firewall chatbots展开，提供了可运行的实现，对理解相关安全机制、工具原理与攻防技术具有较高的研究价值。

从技术实现角度看，此类项目的核心在于对底层原理的深入理解与工程化落地。开发者在研究过程中可以结合官方文档与社区讨论，梳理项目的初始化流程、核心模块职责与数据流转，逐步掌握其中的技术细节与设计思路。

使用与学习建议

1. 下载源码压缩包，解压到本地环境。

2. 根据项目类型准备对应的运行环境与依赖。

3. 按照 README 中的步骤完成配置与部署。

4. 在隔离的测试环境中运行验证，避免影响生产系统。

免责声明

本资源来源于开源社区，仅供安全研究、技术学习与合法授权范围内的攻防演练使用，请勿用于任何非法或侵害他人权益的用途。使用本项目产生的一切后果由使用者自行承担。