OWASP Foundation Web Respository

Related tags

Miscellaneouswwwgrep
Overview

WWWGrep

OWASP Foundation Web Respository

Author: Mark Deen & Aditi Mohan

Introduction

WWWGrep is a rapid search “grepping” mechanism that examines HTML elements by type and permits focused (single), multiple (file based URLs) and recursive (with respect to root domain or not) searches to be performed. Header names and values may also be recursively searched in this manner. WWWGrep was designed to help both breakers and builders to quickly examine code bases under inspection, some use cases and examples are shown below.

Installation

git clone 
pip3 install -r requirements.txt
python3 wwwgrep.py <arguments and parameters>

Dependencies (pip3 install -r requirements.txt)

- Python 3.5+
- BeautifulSoup 4 
- UrlLib.parse
- requests_html
- argparse
- requests
- re
- os.path

Breakers

  • Quickly locate login pages by searching for input fields named “username” or “password” on a site an using a recursion flag
  • Quickly check headers for the use of specific technologies
  • Quickly locate cookies and JWT tokens by search response headers
  • Use with a proxy tool to automate recursion through a set links rapidly
  • Locate all input sinks on a page (or site) by search for input fields and parameter processing symbology
  • Locate all developer comments on a page to identify commented out code (or To Do’s)
  • Quickly test consistency of site controls implemented during recursion (headers, HSTS, CSP etc)
  • Quickly find vulnerable JavaScript code present in web pages
  • Identify API tokens and access keys present in page code

Builders

  • Quickly test multiple sites under management for the use of vulnerable code
  • Quickly test multiple sites under management for the use of vulnerable frameworks/technologies
  • Find sites which may share a common codebase to determine the impact of flaws/vulnerabilities
  • Find sites which share a common authentication token (header auth token)
  • Find sites which may contain developer comments for server hygiene purposes

Command line switches

wwwgrep.py [target/file] [search_string] [search params/criteria/recursion etc]
Search Inputs

search_string		Specify the string to search for or alternatively “” 
			for all objects of type specified in search parameters

-t	--target	Specify a single URL as a target for the search
-f	--file		Specify a file containing a list of URLs to search

Recursion

-rr	--recurse-root	Limits URL recursion to the domain provided in the target
-ra	--recurse-any	Allows recursion to extend beyond the domain of the target

Matching Criteria

-i	--ignore-case	Performs case insensitive matching (default is to respect case)
-d	--dedupe        Allow duplicate findings per page (default is to de-duplicate findings)
-r	--no-redirects	Do not allow redirects (default is to allow redirects)
-b	--no-base-url   Omit the URL of the match from the output (default is to include the URL)
-x	--regex         Allows the use of RegEX matches (search_string is treated as a RegEX, default is off) 
-e	--separator	Specify and output specifier (default is : ) 
-j	--java-render   Turns on JavaScript rendering of page objects and text (default is off) 
-p	--linked-js-on  Turns on searching of linked (script src tags) Java Script (default is off)

Request Parameters

-ps	--https-proxy	Specify a proxy for the HTTPS protocol in https://<ip>:<port> format
-pp 	--http-proxy	Specify a proxy for the HTTP protocol in http://<ip>:<port> format
-hu	--user-agent	Specify a string to use as the user agent in the request
-ha	--auth-header	Specify a bearer token or other auth string to use in the request header

Search Parameters

-s	--all		Search all page HTML and scripts for terms that match the search specification
-sr	--relative	Search page links that match the search specification as relative URLs
-sa	--absolute	Search page links that match the search specification as absolute URLs
-si	--input-fields	Search page input fields that match the search specification
-ss	--scripts	Search scripts tags that match the search specification
-st	--text          Search visible text on the page that matches the search specification
-sc	--comments      Search comments on the page that match the search specification
-sm	--meta          Search in page metadata for matches to the search specification
-sf	--hidden        Search in hidden fields for specific matches to the search specification
-sh	--header-name	Search response headers for specific matches to the search specification
-sv	--header-value  Search response header values for specific matches to the search specification

Examples of use:

Find all input fields named login on a site recursively while not leaving the root domain without case sensitivity in the match

wwwgrep.py -t https://www.target.com -i -si “login” -rr

Find all comments containing the term “to do” on all pages in a site

wwwgrep.py -t https://www.target.com -i -sc “to do” -rr

Find all comments on a specific web page

wwwgrep.py -t https://www.target.com/some_page -i -sc “”

Find all hidden fields within a list of web applications contained in the file input.txt using site recursion

wwwgrep.py -f input.txt -sf “” -rr

Owner
OWASP
The OWASP Foundation
OWASP
Yakuake session management

yman is a python script used for saving/restoring yakuake sessions (currently running commands, working directories, environment variables, tab titles)

Szymon Borecki 6 Jun 25, 2022
Python tools for experimenting with differentiable intonation cost measures

Differentiable Intonation Tools The Differentiable Intonation Tools (dit) are a collection of Python functions to analyze the intonation in multitrack

Simon Schwär 2 Mar 27, 2022
Sudo type me a payload

payloadSecretary Sudo type me a payload Have you ever found yourself having to perform a test, and a client has provided you with a VM inside a VDI in

7 Jul 21, 2022
A repo to record how I prepare my Interview, and really hope it can help you as well. Really appreciate Kieran's help in the pattern's part.

Project Overview The purpose of this repo is to help others to find solutions and explaintion I will commit a solution and explanation to every proble

Vincent Zhenhao ZHAO 1 Nov 29, 2021
Extra scripts to improve user experience related to OpenTaiko

OpenTaiko-Utils Extra scripts to improve user experience related to OpenTaiko osu2tja /!\ IMPORTANT NOTE /!\ Converted charts that aren't yours are fo

2 Dec 25, 2022
Simple kivy project to help new kivy users build android apps with python.

Kivy Calculator A Simple Calculator made with kivy framework.Works on all platforms from Windows/linux to android. Description Simple kivy project to

Oussama Ben Sassi 6 Oct 06, 2022
Get a list of content on your Netflix My List that is expiring in the next month or two.

Netflix My List Expiring Movies Annoyed at Netflix for taking away your movies? Now you don't have to be! Installation instructions Install selenium C

24 Aug 06, 2022
Meaningful and minimalist release notes for developers

Managing manual release notes is hard. Therefore, everyone tends to generate release notes from commit messages. But, you won't get a meaningful release note at the end.

codezri 31 Dec 30, 2022
Poetry workspace plugin for Python monorepos.

poetry-workspace-plugin Poetry workspace plugin for Python monorepos. Inspired by Yarn Workspaces. Adds a new subcommand group, poetry workspace, whic

Jack Smith 74 Jan 01, 2023
Just imagine normal bancho, but you can have multiple profiles and funorange speed up maps ranked

Local osu! server Just imagine normal bancho, but you can have multiple profiles and funorange speed up maps ranked (coming soon)! Windows Setup Insta

Cover 25 Nov 15, 2022
Checkers Project Built Using Python

Checkers Project Built Using Python

Meekness Anyaeche 1 Nov 08, 2021
Tiny demo site for exploring SameSite=Lax

samesite-lax-demo Background on my blog: Exploring the SameSite cookie attribute for preventing CSRF This repo holds some tools for exploring the impl

Simon Willison 6 Nov 10, 2021
Scalene: a high-performance, high-precision CPU, GPU, and memory profiler for Python

Scalene: a high-performance CPU, GPU and memory profiler for Python by Emery Berger, Sam Stern, and Juan Altmayer Pizzorno. Scalene community Slack Ab

PLASMA @ UMass 7k Dec 30, 2022
A example project's description is a high-level overview of why you’re doing a project.

A example project's description is a high-level overview of why you’re doing a project.

Nikita Matyukhin 12 Mar 23, 2022
A simple, light-weight and highly maintainable online judge system for secondary education

y³OJ a simple, light-weight and highly maintainable online judge system for secondary education 一个简单、轻量化、易于维护的、为中学信息技术学科课业教学设计的 Online Judge 系统。 Onlin

20 Oct 04, 2022
resultados (data) de elecciones 2021 y código para extraer data de la ONPE

elecciones-peru-2021-ONPE Resultados (data) de elecciones 2021 y código para extraer data de la ONPE Data Licencia liberal, pero si vas a usarlo por f

Ragi Yaser Burhum 21 Jun 14, 2021
3x+1 recreated in Python

3x-1 3x+1 recreated in Python If a number is odd it is multiplied by 3 and 1 is added to the product. If a number is even it is divided by 2. These ru

4 Aug 19, 2022
A python script that will automate the boring task of login to the captive portal again and again

A python script that will automate the boring task of login to the captive portal again and again

Rakib Hasan 2 Feb 09, 2022
Python Project Template

A low dependency and really simple to start project template for Python Projects.

Bruno Rocha 651 Dec 29, 2022
A jokes python module

Made with Python3 (C) @FayasNoushad Copyright permission under MIT License License - https://github.com/FayasNoushad/Jokes/blob/main/LICENSE Deploy

Fayas Noushad 3 Nov 28, 2021