OWASP Foundation Web Respository

Related tags

Miscellaneouswwwgrep
Overview

WWWGrep

OWASP Foundation Web Respository

Author: Mark Deen & Aditi Mohan

Introduction

WWWGrep is a rapid search “grepping” mechanism that examines HTML elements by type and permits focused (single), multiple (file based URLs) and recursive (with respect to root domain or not) searches to be performed. Header names and values may also be recursively searched in this manner. WWWGrep was designed to help both breakers and builders to quickly examine code bases under inspection, some use cases and examples are shown below.

Installation

git clone 
pip3 install -r requirements.txt
python3 wwwgrep.py <arguments and parameters>

Dependencies (pip3 install -r requirements.txt)

- Python 3.5+
- BeautifulSoup 4 
- UrlLib.parse
- requests_html
- argparse
- requests
- re
- os.path

Breakers

  • Quickly locate login pages by searching for input fields named “username” or “password” on a site an using a recursion flag
  • Quickly check headers for the use of specific technologies
  • Quickly locate cookies and JWT tokens by search response headers
  • Use with a proxy tool to automate recursion through a set links rapidly
  • Locate all input sinks on a page (or site) by search for input fields and parameter processing symbology
  • Locate all developer comments on a page to identify commented out code (or To Do’s)
  • Quickly test consistency of site controls implemented during recursion (headers, HSTS, CSP etc)
  • Quickly find vulnerable JavaScript code present in web pages
  • Identify API tokens and access keys present in page code

Builders

  • Quickly test multiple sites under management for the use of vulnerable code
  • Quickly test multiple sites under management for the use of vulnerable frameworks/technologies
  • Find sites which may share a common codebase to determine the impact of flaws/vulnerabilities
  • Find sites which share a common authentication token (header auth token)
  • Find sites which may contain developer comments for server hygiene purposes

Command line switches

wwwgrep.py [target/file] [search_string] [search params/criteria/recursion etc]
Search Inputs

search_string		Specify the string to search for or alternatively “” 
			for all objects of type specified in search parameters

-t	--target	Specify a single URL as a target for the search
-f	--file		Specify a file containing a list of URLs to search

Recursion

-rr	--recurse-root	Limits URL recursion to the domain provided in the target
-ra	--recurse-any	Allows recursion to extend beyond the domain of the target

Matching Criteria

-i	--ignore-case	Performs case insensitive matching (default is to respect case)
-d	--dedupe        Allow duplicate findings per page (default is to de-duplicate findings)
-r	--no-redirects	Do not allow redirects (default is to allow redirects)
-b	--no-base-url   Omit the URL of the match from the output (default is to include the URL)
-x	--regex         Allows the use of RegEX matches (search_string is treated as a RegEX, default is off) 
-e	--separator	Specify and output specifier (default is : ) 
-j	--java-render   Turns on JavaScript rendering of page objects and text (default is off) 
-p	--linked-js-on  Turns on searching of linked (script src tags) Java Script (default is off)

Request Parameters

-ps	--https-proxy	Specify a proxy for the HTTPS protocol in https://<ip>:<port> format
-pp 	--http-proxy	Specify a proxy for the HTTP protocol in http://<ip>:<port> format
-hu	--user-agent	Specify a string to use as the user agent in the request
-ha	--auth-header	Specify a bearer token or other auth string to use in the request header

Search Parameters

-s	--all		Search all page HTML and scripts for terms that match the search specification
-sr	--relative	Search page links that match the search specification as relative URLs
-sa	--absolute	Search page links that match the search specification as absolute URLs
-si	--input-fields	Search page input fields that match the search specification
-ss	--scripts	Search scripts tags that match the search specification
-st	--text          Search visible text on the page that matches the search specification
-sc	--comments      Search comments on the page that match the search specification
-sm	--meta          Search in page metadata for matches to the search specification
-sf	--hidden        Search in hidden fields for specific matches to the search specification
-sh	--header-name	Search response headers for specific matches to the search specification
-sv	--header-value  Search response header values for specific matches to the search specification

Examples of use:

Find all input fields named login on a site recursively while not leaving the root domain without case sensitivity in the match

wwwgrep.py -t https://www.target.com -i -si “login” -rr

Find all comments containing the term “to do” on all pages in a site

wwwgrep.py -t https://www.target.com -i -sc “to do” -rr

Find all comments on a specific web page

wwwgrep.py -t https://www.target.com/some_page -i -sc “”

Find all hidden fields within a list of web applications contained in the file input.txt using site recursion

wwwgrep.py -f input.txt -sf “” -rr

Owner
OWASP
The OWASP Foundation
OWASP
Set of tools to analyze Tinynuke samples

tinynuke-toolset You'll find in that repository a set of tools and scripts I developped to analyze Tinynuke samples. Dll extractor: script used to ext

Heat Miser 14 Aug 18, 2022
A simple weather app.

keather A simple weather app. This is currently not finished. Dependencies: yay -S python-beautifulsoup4 tk

1 Jan 09, 2022
Perform oocyst segmentation in mercurochrome stained mosquito midgut

Midgut_oocyst_segmentation Perform oocyst segmentation in mercurochrome stained mosquito midguts This oocyst segmentation model also powers the webtoo

Duo Peng 3 Oct 27, 2021
Introduction to Databases Coursework 2 (SQL) - dataset generator

Introduction to Databases Coursework 2 (SQL) - dataset generator This is python script generates a text file with insert queries for the schema.sql fi

Javier Bosch 1 Nov 08, 2021
Python communism - A module for initiating the communist revolution in each of our python modules

Python communist revolution A man once said to abolish the classes or something

758 Jan 03, 2023
Tiny demo site for exploring SameSite=Lax

samesite-lax-demo Background on my blog: Exploring the SameSite cookie attribute for preventing CSRF This repo holds some tools for exploring the impl

Simon Willison 6 Nov 10, 2021
「📖」Tool created to extract metadata from a domain

Metafind is an OSINT tool created with the aim of automating the search for metadata of a particular domain from the search engine known as Google.

9 Dec 28, 2022
Simple Python tool to check if there is an Office 365 instance linked to a domain.

o365chk.py Simple Python script to check if there is an Office365 instance linked to a particular domain.

Steven Harris 37 Jan 02, 2023
Flight Reservation App With Python

Flight Reservation App With Python

victor-h. 1 Nov 21, 2021
It is Keqin Wang first project in CMU, trying to use DRL(PPO) to control a 5-dof manipulator to draw line in space.

5dof-robot-writing this project aim to use PPO control a 5 dof manipulator to draw lines in 3d space. Introduction to the files the pybullet environme

Keqin Wang 4 Aug 22, 2022
Entitlement AND Hardened Runtime Check

Python3 script for macOS to recursively check /Applications and also check /usr/local/bin, /usr/bin, and /usr/sbin for binaries with problematic/interesting entitlements. Also checks for hardened run

Cedric Owens 79 Nov 16, 2022
tagls is a language server based on gtags.

tagls tagls is a language server based on gtags. Why I wrote it? Almost all modern editors have great support to LSP, but language servers based on se

daquexian 31 Dec 01, 2022
API Rate Limit Decorator

ratelimit APIs are a very common way to interact with web services. As the need to consume data grows, so does the number of API calls necessary to re

Tomas Basham 574 Dec 26, 2022
Multi-Process / Censorship Detection

Multi-Process / Censorship Detection

Baris Dincer 2 Dec 22, 2021
Jarvis Python BOT acts like Google-assistance

Jarvis-Python-BOT Jarvis Python BOT acts like Google-assistance Setup Add Mail ID (Gmail) in the file at line no 82.

Ishan Jogalekar 1 Jan 08, 2022
A simple IDA Pro plugin to show all HexRays decompiler comments written by user

XRaysComments A simple IDA Pro plugin to show all HexRays decompiler comments written by user Installation Copy the file xray_comments.py to the plugi

Nox 20 Dec 27, 2022
1cak - An Indonesian web that provide lot of fun.

An unofficial API of 1cak.com 1cak - An Indonesian web that provide lot of fun. Endpoint Lol - 10 Recent stored posts on database Example: https://on

Dicky Mulia Fiqri 5 Sep 27, 2022
MDAnalysis tool to calculate membrane curvature.

The MDAkit for membrane curvature analysis is part of the Google Summer of Code program and it is linked to a Code of Conduct.

MDAnalysis 19 Oct 20, 2022
TimeWizard - A script that generates every single Time Wizard EDOPRO lflist possible

EDOPRO F&L list generator This project is just a script that generates every sin

Diamond Dude 2 Sep 28, 2022
WGGCommute - Adding Commute Times to WG-Gesucht Listings

WGGCommute - Adding Commute Times to WG-Gesucht Listings This is a barebones implementation of a chrome extension that can be used to add commute time

Jannis 2 Jul 20, 2022