gitlab-watchman - Finding exposed secrets and personal data in GitLa

项目说明

<img src=" width="550">

GitLab Watchman

!Python 2.7 and 3 compatible
!PyPI
!License: MIT

About GitLab Watchman

GitLab Watchman is an application that uses the GitLab API to detect exposed secrets and personal data. It also enumerates the GitLab instance for any useful information.

Features

#### Secrets Detection
It searches GitLab for internally shared projects and looks at:

Code

Commits

Wiki pages

Issues

Merge requests

Milestones

Notes

Snippets

For the following data:

GCP keys and service account files

AWS keys

Azure keys and service account files

Google API keys

Slack API tokens & webhooks

Private keys (SSH, PGP, any other misc private key)

Exposed tokens (Bearer tokens, access tokens, client_secret etc.)

S3 config files

Tokens for services such as Heroku, PayPal and more

Passwords in plaintext

and more

##### Time based searching
You can run GitLab Watchman to look for results going back as far as:

24 hours

7 days

30 days

All time

This means after one deep scan, you can schedule GitLab Watchman to run regularly and only return results from your chosen timeframe.

#### Enumeration
GitLab Watchman can enumerate potentially useful information from a GitLab instance:

Instance metadata

Information on the calling user/token being used

Output all users to CSV file

Output all projects to CSV file

Output all groups to CSV file

Signatures

GitLab Watchman uses custom YAML signatures to detect matches in GitLab. These signatures are pulled from the central Watchman Signatures repository. Slack Watchman automatically updates its signature base at runtime to ensure its using the latest signatures to detect secrets.

#### Suppressing Signatures
You can define signatures that you want to disable when running GitLab Watchman by adding their IDs to the disabled_signatures section of the watchman.conf file. For example:

You can find the ID of a signature in the indi YAML files in Watchman Signatures repository.

Logging

GitLab Watchman gives the following logging options:

Terminal-friendly Stdout

JSON to Stdout

GitLab Watchman defaults to terminal-friendly stdout logging if no option is given. This is designed to be easier for humans to read.

JSON logging is also a, which is perfect for ingesting into a SIEM or other log analysis platforms.

JSON formatted logging can be easily redirected to a file as below:

Requirements

GitLab

GitLab Watchman uses the v4 API, and works with GitLab Enterprise Edition :

13.0 and above - Yes

GitLab.com - Yes

12.0 - 12.10 - Maybe, untested but if using v4 of the API then it could work

GitLab Licence & Elasticsearch

To search the scopes:

blobs

wiki_blobs

commits

The GitLab instance must have Elasticsearch configured, and be running Enterprise Edition with a minimum GitLab Starter or Bronze Licence.

GitLab personal access token

To run GitLab Watchman, you will need a GitLab personal access token.

You can create a personal access token in the GitLab GUI -> Access Tokens -> Add a personal access token

The token needs permission for the following scopes:

Note: Personal access tokens act on behalf of the user who creates them, so I would suggest you create a token using a service account, otherwise the app will ha to your private repositories.

GitLab URL

You also need to provide the URL of your GitLab instance.

#### Pro token & URL
GitLab Watchman will get the GitLab token and URL from the en GITLAB_WATCHMAN_TOKEN and GITLAB_WATCHMAN_URL.

watchman.conf file

Configuration options can be passed in a file named watchman.conf which must be stored in your home directory. The file should follow the YAML format, and should look like below:

GitLab Watchman will look for this file at runtime, and use the configuration options from here.

Installation

You can install the latest stable via pip:

python3 -m pip install gitlab-watchman

Or build from source yourself. Download the release source files, then from the top le run:

Docker Image

GitLab Watchman is also a from the Docker hub as a Docker image:

docker pull papermountain/gitlab-watchman:latest

You can then run GitLab Watchman in a container, making sure you pass the required en :

Usage

GitLab Watchman will be installed as a global command, use as follows:

Other Watchman apps

You may be interested in the other apps in the Watchman family:

Slack Watchman

Slack Watchman for Enterprise Grid

GitHub Watchman

License

The source code for this project is released under the GNU General Public Licence. This project is not associated with GitLab.

项目信息

开发语言：Python

Star 数：206

技术要点

该项目属于红蓝攻防方向的开源资源，代码结构清晰，适合安全研究人员学习参考。项目围绕Finding exposed secrets and personal data in GitLab展开，提供了可运行的实现，对理解相关安全机制、工具原理与攻防技术具有较高的研究价值。

从技术实现角度看，此类项目的核心在于对底层原理的深入理解与工程化落地。开发者在研究过程中可以结合官方文档与社区讨论，梳理项目的初始化流程、核心模块职责与数据流转，逐步掌握其中的技术细节与设计思路。

使用与学习建议

1. 下载源码压缩包，解压到本地环境。

2. 根据项目类型准备对应的运行环境与依赖。

3. 按照 README 中的步骤完成配置与部署。

4. 在隔离的测试环境中运行验证，避免影响生产系统。

免责声明

本资源来源于开源社区，仅供安全研究、技术学习与合法授权范围内的攻防演练使用，请勿用于任何非法或侵害他人权益的用途。使用本项目产生的一切后果由使用者自行承担。