OWASP Foundation Web Respository

Related tags

Miscellaneouswwwgrep
Overview

WWWGrep

OWASP Foundation Web Respository

Author: Mark Deen & Aditi Mohan

Introduction

WWWGrep is a rapid search “grepping” mechanism that examines HTML elements by type and permits focused (single), multiple (file based URLs) and recursive (with respect to root domain or not) searches to be performed. Header names and values may also be recursively searched in this manner. WWWGrep was designed to help both breakers and builders to quickly examine code bases under inspection, some use cases and examples are shown below.

Installation

git clone 
pip3 install -r requirements.txt
python3 wwwgrep.py <arguments and parameters>

Dependencies (pip3 install -r requirements.txt)

- Python 3.5+
- BeautifulSoup 4 
- UrlLib.parse
- requests_html
- argparse
- requests
- re
- os.path

Breakers

  • Quickly locate login pages by searching for input fields named “username” or “password” on a site an using a recursion flag
  • Quickly check headers for the use of specific technologies
  • Quickly locate cookies and JWT tokens by search response headers
  • Use with a proxy tool to automate recursion through a set links rapidly
  • Locate all input sinks on a page (or site) by search for input fields and parameter processing symbology
  • Locate all developer comments on a page to identify commented out code (or To Do’s)
  • Quickly test consistency of site controls implemented during recursion (headers, HSTS, CSP etc)
  • Quickly find vulnerable JavaScript code present in web pages
  • Identify API tokens and access keys present in page code

Builders

  • Quickly test multiple sites under management for the use of vulnerable code
  • Quickly test multiple sites under management for the use of vulnerable frameworks/technologies
  • Find sites which may share a common codebase to determine the impact of flaws/vulnerabilities
  • Find sites which share a common authentication token (header auth token)
  • Find sites which may contain developer comments for server hygiene purposes

Command line switches

wwwgrep.py [target/file] [search_string] [search params/criteria/recursion etc]
Search Inputs

search_string		Specify the string to search for or alternatively “” 
			for all objects of type specified in search parameters

-t	--target	Specify a single URL as a target for the search
-f	--file		Specify a file containing a list of URLs to search

Recursion

-rr	--recurse-root	Limits URL recursion to the domain provided in the target
-ra	--recurse-any	Allows recursion to extend beyond the domain of the target

Matching Criteria

-i	--ignore-case	Performs case insensitive matching (default is to respect case)
-d	--dedupe        Allow duplicate findings per page (default is to de-duplicate findings)
-r	--no-redirects	Do not allow redirects (default is to allow redirects)
-b	--no-base-url   Omit the URL of the match from the output (default is to include the URL)
-x	--regex         Allows the use of RegEX matches (search_string is treated as a RegEX, default is off) 
-e	--separator	Specify and output specifier (default is : ) 
-j	--java-render   Turns on JavaScript rendering of page objects and text (default is off) 
-p	--linked-js-on  Turns on searching of linked (script src tags) Java Script (default is off)

Request Parameters

-ps	--https-proxy	Specify a proxy for the HTTPS protocol in https://<ip>:<port> format
-pp 	--http-proxy	Specify a proxy for the HTTP protocol in http://<ip>:<port> format
-hu	--user-agent	Specify a string to use as the user agent in the request
-ha	--auth-header	Specify a bearer token or other auth string to use in the request header

Search Parameters

-s	--all		Search all page HTML and scripts for terms that match the search specification
-sr	--relative	Search page links that match the search specification as relative URLs
-sa	--absolute	Search page links that match the search specification as absolute URLs
-si	--input-fields	Search page input fields that match the search specification
-ss	--scripts	Search scripts tags that match the search specification
-st	--text          Search visible text on the page that matches the search specification
-sc	--comments      Search comments on the page that match the search specification
-sm	--meta          Search in page metadata for matches to the search specification
-sf	--hidden        Search in hidden fields for specific matches to the search specification
-sh	--header-name	Search response headers for specific matches to the search specification
-sv	--header-value  Search response header values for specific matches to the search specification

Examples of use:

Find all input fields named login on a site recursively while not leaving the root domain without case sensitivity in the match

wwwgrep.py -t https://www.target.com -i -si “login” -rr

Find all comments containing the term “to do” on all pages in a site

wwwgrep.py -t https://www.target.com -i -sc “to do” -rr

Find all comments on a specific web page

wwwgrep.py -t https://www.target.com/some_page -i -sc “”

Find all hidden fields within a list of web applications contained in the file input.txt using site recursion

wwwgrep.py -f input.txt -sf “” -rr

Owner
OWASP
The OWASP Foundation
OWASP
Shows VRML team stats of all players in your pubs

VRML Team Stat Searcher Displays Team Name, Team Rank (Worldwide), and tier of all the players in your pubs. GUI WIP: Only username search works (for

Hamish Burke 2 Dec 22, 2022
K2HASH Python library - NoSQL Key Value Store(KVS) library

k2hash_python Overview k2hash_python is an official python driver for k2hash. Install Firstly you must install the k2hash shared library: curl -o- htt

Yahoo! JAPAN 3 Oct 19, 2022
Anonfiles files leaker via keyword.

Anonfiles files leaker via keyword

Trac3D1y 6 Nov 23, 2022
The purpose of this code base is to add a specified signal-to-noise ratio noise from MUSAN dataset to a pure speech signal and to generate far-field speech data using room impulse response data from BUT [email protected] Reverb Database.

Add_noise_and_rir_to_speech The purpose of this code base is to add a specified signal-to-noise ratio noise from MUSAN dataset to a pure speech signal

Yunqi Chen 7 Oct 30, 2022
Diff Match Patch is a high-performance library in multiple languages that manipulates plain text.

The Diff Match and Patch libraries offer robust algorithms to perform the operations required for synchronizing plain text. Diff: Compare two blocks o

Google 5.9k Dec 30, 2022
A fishing bot script written in Python!

A fishing bot script written in Python!

Anel Drocic 3 Nov 03, 2021
a really simple bot that send you memes from reddit to whatsapp

a really simple bot that send you memes from reddit to whatsapp want to use use it? install the dependencies with pip3 install -r requirements.txt the

pai 10 Nov 28, 2021
AIST++ API This repo contains starter code for using the AIST++ dataset.

Explainability for Vision Transformers (in PyTorch) This repository implements methods for explainability in Vision Transformers

Google 260 Dec 30, 2022
Calculadora-basica - Calculator with basic operators

Calculadora básica Calculadora com operadores básicos; O programa solicitará a d

Vitor Antoni 2 Apr 26, 2022
Procedurally generated Oblique Strategies for writing your own Oblique Strategies

Procedurally generated Oblique Strategies for writing your own Oblique Strategies.

Gordon Brander 13 Aug 17, 2022
System Design Assignments as part of Arpit's System Design Masterclass

System Design Assignments The repository contains a set of problem statements around Software Architecture and System Design as conducted by Arpit's S

Relog 1.1k Jan 09, 2023
Repositório do programa ConstruDelas - Trilha Python - Módulos 1 e 2

ConstruDelas - Introdução ao Python Nome: Visão Geral Bem vinda ao repositório do curso ConstruDelas, módulo de Introdução ao Python. Aqui vamos mante

WoMakersCode 8 Oct 14, 2022
Senator Stock Trading Tester

Senator Stock Trading Tester Program to compare stock performance of Senator's transactions vs when the sale is disclosed. Using to find if tracking S

Cole Cestaro 1 Dec 07, 2021
Grammar of Scalable Linked Interactive Nucleotide Graphics

Gosling.js Gosling.js is a declarative grammar for interactive (epi)genomics visualization on the Web. ⚠️ Please be aware that the grammar of Gosling.

Gosling 126 Nov 29, 2022
Consolemenu on python with pynput

ConsoleMenu Consolemenu on python 3 with pynput Powered by pynput and colorama Description Модуль позволяющий сделать меню выбора с помощью стрелок дл

KrouZ_CZ 2 Nov 15, 2021
Retrieve bank transactions and categorize for budgeting use

Budgeting After trying out some budgeting software, I decided to make my own. selenium_scraper Using the selenium package, this script runs an instanc

Marc 1 Nov 10, 2021
Stock Monitoring

Stock Monitoring Description It is a stock monitoring script. This repository is still under developing. Getting Started Prerequisites & Installing pi

Sission 1 Feb 03, 2022
Dapp / Forge traces enhancer

traces-explorer Dapp / Forge traces enhancer Usage traces.py and pattern_* files should be in the same directory make test traces.txt py traces.

1 Feb 02, 2022
A collection of existing KGQA datasets in the form of the huggingface datasets library, aiming to provide an easy-to-use access to them.

KGQA Datasets Brief Introduction This repository is a collection of existing KGQA datasets in the form of the huggingface datasets library, aiming to

Semantic Systems research group 21 Jan 06, 2023
jmespath.rs Python binding

rjmespath-py jmespath.rs Python binding.

messense 3 Dec 14, 2022